Bug 174526

Summary: main website (freebsd.org/www.freebsd.org) is unreachable from Nova Scotia Universities
Product: Documentation Reporter: Joseph Mingrone <jrm>
Component: Books & ArticlesAssignee: freebsd-doc (Nobody) <doc>
Status: Closed FIXED    
Severity: Affects Only Me    
Priority: Normal    
Version: Latest   
Hardware: Any   
OS: Any   

Description Joseph Mingrone freebsd_committer freebsd_triage 2012-12-17 20:40:00 UTC 
	For the past month or so, freebsd.org has been unreachable from Dalhousie University (Halifax, Nova Scotia, Canada) IP addresses.  Until today, www.freebsd.org was accessible.  The traceroute (pasted below) shows problems after routerer-pat1.freebsd.org (216.115.101.225). Connecting from IPs from local, private ISPs works fine.  Perhaps there is an invalid redirect for Atlantic Canadian university IP blocks?

% traceroute www.freebsd.org
traceroute to wfe0.ysv.freebsd.org (8.8.178.110), 64 hops max, 52 byte packets
 1  GW81AD2000.Backbone.Dal.Ca (129.173.32.1)  0.870 ms  0.551 ms  0.480 ms
 2  GigaPOP-gw.acorn-ns.Ca (198.166.1.17)  0.846 ms  0.758 ms  0.792 ms
 3  ISInet-GW.Backbone.Dal.Ca (198.166.1.38)  0.769 ms  0.740 ms  0.726 ms
 4  ns-hlfx-asr002.ns.eastlink.ca (24.222.79.121)  1.261 ms  1.232 ms  0.925 ms
 5  ns-hlfx-dr001.ns.eastlink.ca (24.215.101.245)  1.766 ms  1.638 ms  1.842 ms
 6  ns-hlfx-br001.ns.eastlink.ca (24.215.101.221)  1.890 ms  1.774 ms  1.727 ms
 7  xe-1-3-0.bos11.ip4.tinet.net (173.241.128.145)  9.129 ms  51.961 ms  16.959 ms
 8  xe-5-1-0.sjc10.ip4.tinet.net (89.149.185.161)  87.429 ms  86.623 ms
    xe-9-1-0.sjc10.ip4.tinet.net (89.149.184.53)  94.843 ms
 9  * * ge-0-3-8.pat1.sjc.yahoo.com (216.115.96.6)  99.273 ms
10  routerer-pat1.freebsd.org (216.115.101.225)  99.996 ms  99.838 ms  98.871 ms
11  routerer-pat1.freebsd.org (216.115.101.225)  98.868 ms  99.718 ms *
12  * * *
13  * * *
...
64  * * *
Comment 1 Simon L. B. Nielsen freebsd_committer freebsd_triage 2012-12-17 21:55:04 UTC 
On 17 Dec 2012, at 20:29, Joseph Mingrone <jrm@ftfl.ca> wrote:

>=20
>> Number:         174526
>> Category:       www
>> Synopsis:       main website (freebsd.org/www.freebsd.org) is =
unreachable from Nova Scotia Universities

> 	For the past month or so, freebsd.org has been unreachable from =
Dalhousie University (Halifax, Nova Scotia, Canada) IP addresses.  Until =
today, www.freebsd.org was accessible.  The traceroute (pasted below) =
shows problems after routerer-pat1.freebsd.org (216.115.101.225). =
Connecting from IPs from local, private ISPs works fine.  Perhaps there =
is an invalid redirect for Atlantic Canadian university IP blocks?
>=20
> % traceroute www.freebsd.org
> traceroute to wfe0.ysv.freebsd.org (8.8.178.110), 64 hops max, 52 byte =
packets
> 1  GW81AD2000.Backbone.Dal.Ca (129.173.32.1)  0.870 ms  0.551 ms  =
0.480 ms
> 2  GigaPOP-gw.acorn-ns.Ca (198.166.1.17)  0.846 ms  0.758 ms  0.792 ms
> 3  ISInet-GW.Backbone.Dal.Ca (198.166.1.38)  0.769 ms  0.740 ms  0.726 =
ms
> 4  ns-hlfx-asr002.ns.eastlink.ca (24.222.79.121)  1.261 ms  1.232 ms  =
0.925 ms
> 5  ns-hlfx-dr001.ns.eastlink.ca (24.215.101.245)  1.766 ms  1.638 ms  =
1.842 ms
> 6  ns-hlfx-br001.ns.eastlink.ca (24.215.101.221)  1.890 ms  1.774 ms  =
1.727 ms
> 7  xe-1-3-0.bos11.ip4.tinet.net (173.241.128.145)  9.129 ms  51.961 ms =
 16.959 ms
> 8  xe-5-1-0.sjc10.ip4.tinet.net (89.149.185.161)  87.429 ms  86.623 ms
>    xe-9-1-0.sjc10.ip4.tinet.net (89.149.184.53)  94.843 ms
> 9  * * ge-0-3-8.pat1.sjc.yahoo.com (216.115.96.6)  99.273 ms
> 10  routerer-pat1.freebsd.org (216.115.101.225)  99.996 ms  99.838 ms  =
98.871 ms
> 11  routerer-pat1.freebsd.org (216.115.101.225)  98.868 ms  99.718 ms =
*
> 12  * * *
> 13  * * *
> ...
> 64  * * *

FreeBSD is in the process of moving from one datacenter to another and I =
switched www.FreeBSD.org to be served from the new one today. The raw =
host "freebsd.org" is freefall.freebsd.org which was moved a bit longer =
ago (1 month sounds right).

So I think it's either a firewall in either end or an odd routing =
problem.

Anyone have obvious candidates ? :

Connecting to www.dal.ca port 80 with an IP from our delegated /24 fails =
with:

# tcpdump -s0 -i bge0 host www.dal.ca
tcpdump: verbose output suppressed, use -v or -vv for full protocol =
decode
listening on bge0, link-type EN10MB (Ethernet), capture size 65535 bytes
21:49:29.941767 IP routerer.FreeBSD.org.41224 > =
kil-cq5-4w0.ITS.Dal.Ca.http: Flags [S], seq 997436128, win 65535, =
options [mss 1460,nop,wscale 6,sackOK,TS val 270875695 ecr 0], length 0
21:49:32.940893 IP routerer.FreeBSD.org.41224 > =
kil-cq5-4w0.ITS.Dal.Ca.http: Flags [S], seq 997436128, win 65535, =
options [mss 1460,nop,wscale 6,sackOK,TS val 270878695 ecr 0], length 0
21:49:36.140885 IP routerer.FreeBSD.org.41224 > =
kil-cq5-4w0.ITS.Dal.Ca.http: Flags [S], seq 997436128, win 65535, =
options [mss 1460,nop,wscale 6,sackOK,TS val 270881895 ecr 0], length 0
21:49:39.340885 IP routerer.FreeBSD.org.41224 > =
kil-cq5-4w0.ITS.Dal.Ca.http: Flags [S], seq 997436128, win 65535, =
options [mss 1460], length 0
21:49:42.540895 IP routerer.FreeBSD.org.41224 > =
kil-cq5-4w0.ITS.Dal.Ca.http: Flags [S], seq 997436128, win 65535, =
options [mss 1460], length 0
21:49:45.740890 IP routerer.FreeBSD.org.41224 > =
kil-cq5-4w0.ITS.Dal.Ca.http: Flags [S], seq 997436128, win 65535, =
options [mss 1460], length 0
21:49:51.940891 IP routerer.FreeBSD.org.41224 > =
kil-cq5-4w0.ITS.Dal.Ca.http: Flags [S], seq 997436128, win 65535, =
options [mss 1460], length 0

But if I'm using the IP on our transit network it works just fine:

21:50:58.046045 IP routerer-pat1.freebsd.org.35814 > =
kil-cq5-4w0.ITS.Dal.Ca.http: Flags [S], seq 2306026374, win 65535, =
options [mss 1460,nop,wscale 6,sackOK,TS val 270963800 ecr 0], length 0
21:50:58.155918 IP kil-cq5-4w0.ITS.Dal.Ca.http > =
routerer-pat1.freebsd.org.35814: Flags [S.], seq 4012957193, ack =
2306026375, win 5792, options [mss 1380,sackOK,TS val 416382003 ecr =
270963800,nop,wscale 7], length 0
21:50:58.155963 IP routerer-pat1.freebsd.org.35814 > =
kil-cq5-4w0.ITS.Dal.Ca.http: Flags [.], ack 1, win 1026, options =
[nop,nop,TS val 270963910 ecr 416382003], length 0
21:51:02.746528 IP kil-cq5-4w0.ITS.Dal.Ca.http > =
routerer-pat1.freebsd.org.35814: Flags [S.], seq 4012957193, ack =
2306026375, win 5792, options [mss 1380,sackOK,TS val 416386594 ecr =
270963910,nop,wscale 7], length 0
21:51:02.746562 IP routerer-pat1.freebsd.org.35814 > =
kil-cq5-4w0.ITS.Dal.Ca.http: Flags [.], ack 1, win 1026, options =
[nop,nop,TS val 270968500 ecr 416386594], length 0

Since I already did them... a few traceroutes:

Traceroute from the old one:

[simon@red:~] traceroute www.dal.ca
traceroute to kil-cq5-4w0.ITS.dal.ca (129.173.21.171), 64 hops max, 40 =
byte packets
 1  vl-225.bas-b1.sp1.yahoo.com (69.147.83.3)  0.398 ms  0.289 ms  0.228 =
ms
 2  ge-1-0-1.msr1.sp1.yahoo.com (209.131.32.30)  0.201 ms  0.196 ms  =
0.183 ms
 3  ae-2-d161.pat2.sjc.yahoo.com (216.115.107.62)  0.829 ms
    ae-0-d141.pat2.pao.yahoo.com (216.115.107.50)  0.834 ms
    ae-1-d160.pat1.sjc.yahoo.com (216.115.107.56)  0.946 ms
 4  ae-3.pat1.pao.yahoo.com (216.115.101.34)  0.964 ms  1.184 ms
    ae-3.pat2.swp.yahoo.com (216.115.96.57)  19.416 ms
 5  ae-0.pat1.swp.yahoo.com (216.115.110.43)  19.550 ms  19.346 ms  =
19.526 ms
 6  clgr2rtr2.canarie.ca (205.189.33.197)  108.554 ms  108.667 ms
    vncv1rtr2.canarie.ca (206.81.80.189)  26.030 ms
 7  toro1rtr2.canarie.ca (205.189.33.209)  108.755 ms  108.740 ms  =
109.493 ms
 8  toro1rtr2.canarie.ca (205.189.33.209)  108.802 ms
    mtrl2rtr2.canarie.ca (205.189.33.223)  110.261 ms  109.952 ms
 9  hlfx1rtr2.canarie.ca (199.212.24.84)  116.929 ms
    mtrl2rtr2.canarie.ca (205.189.33.223)  113.690 ms  110.201 ms
10  hlfx1rtr2.canarie.ca (199.212.24.84)  116.991 ms  116.898 ms *
11  * * *

[simon@admin:~] traceroute www.dal.ca
traceroute to kil-cq5-4w0.ITS.dal.ca (129.173.21.171), 64 hops max, 52 =
byte packets
 1  routerer (8.8.178.2)  0.708 ms  0.581 ms  0.675 ms
 2  ae-1-d188.pat1.sjc.yahoo.com (216.115.101.224)  1.818 ms  1.822 ms  =
1.663 ms
 3  ae-6.pat1.swp.yahoo.com (216.115.100.94)  20.470 ms  20.569 ms  =
20.523 ms
 4  * * *
 5  clgr2rtr2.canarie.ca (205.189.33.197)  107.807 ms  107.904 ms  =
107.759 ms
 6  toro1rtr2.canarie.ca (205.189.33.209)  107.981 ms  108.130 ms  =
108.106 ms
 7  mtrl2rtr2.canarie.ca (205.189.33.223)  108.661 ms  108.388 ms  =
108.170 ms
 8  hlfx1rtr2.canarie.ca (199.212.24.84)  107.617 ms  107.620 ms  =
107.279 ms
 9  * * *
10  * * *
11  * * *

Just for reference for me from home to www.freebsd.org the last part is:

10  4.68.111.242 (4.68.111.242)  166.288 ms
    4.68.63.158 (4.68.63.158)  139.719 ms
    64.215.195.213 (64.215.195.213)  149.850 ms
11  ae14-20g.scr3.nyc1.gblx.net (67.17.72.109)  129.934 ms  103.820 ms  =
113.754 ms
12  ae4-20g.scr4.snv2.gblx.net (67.16.147.90)  179.863 ms  393.493 ms
    xe5-0-2-10g.scr3.snv2.gblx.net (67.16.165.50)  238.116 ms
13  e5-3-40g.ar5.sjc2.gblx.net (67.17.72.14)  249.433 ms
    e8-1-20g.ar5.sjc2.gblx.net (67.16.145.118)  422.696 ms  198.738 ms
14  yahoo.tengigabitethernet2-4.1189.ar3.sjc2.gblx.net (208.48.239.254)  =
192.603 ms  198.490 ms
    yahoo-san-jose.tengig2-3.1189.ar3.sjc2.gblx.net (64.211.206.210)  =
638.591 ms
15  routerer-pat1.freebsd.org (216.115.101.225)  663.967 ms  499.559 ms  =
280.826 ms
16  wfe0.ysv.freebsd.org (8.8.178.110)  333.695 ms  482.018 ms  249.999 =
ms

--=20
Simon L. B. Nielsen
Comment 2 Joseph Mingrone freebsd_committer freebsd_triage 2012-12-19 15:53:01 UTC 
On Mon, Dec 17, 2012 at 5:55 PM, Simon L. B. Nielsen <simon@freebsd.org> wrote:
> Connecting to www.dal.ca port 80 with an IP from our delegated /24 fails with:
>
> # tcpdump -s0 -i bge0 host www.dal.ca
> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
> listening on bge0, link-type EN10MB (Ethernet), capture size 65535 bytes
> 21:49:29.941767 IP routerer.FreeBSD.org.41224 > kil-cq5-4w0.ITS.Dal.Ca.http: Flags [S], seq 997436128, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 270875695 ecr 0], length 0
> 21:49:32.940893 IP routerer.FreeBSD.org.41224 > kil-cq5-4w0.ITS.Dal.Ca.http: Flags [S], seq 997436128, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 270878695 ecr 0], length 0
> 21:49:36.140885 IP routerer.FreeBSD.org.41224 > kil-cq5-4w0.ITS.Dal.Ca.http: Flags [S], seq 997436128, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 270881895 ecr 0], length 0
> 21:49:39.340885 IP routerer.FreeBSD.org.41224 > kil-cq5-4w0.ITS.Dal.Ca.http: Flags [S], seq 997436128, win 65535, options [mss 1460], length 0
> 21:49:42.540895 IP routerer.FreeBSD.org.41224 > kil-cq5-4w0.ITS.Dal.Ca.http: Flags [S], seq 997436128, win 65535, options [mss 1460], length 0
> 21:49:45.740890 IP routerer.FreeBSD.org.41224 > kil-cq5-4w0.ITS.Dal.Ca.http: Flags [S], seq 997436128, win 65535, options [mss 1460], length 0
> 21:49:51.940891 IP routerer.FreeBSD.org.41224 > kil-cq5-4w0.ITS.Dal.Ca.http: Flags [S], seq 997436128, win 65535, options [mss 1460], length 0
>

Are you able to connect any one of www.smu.ca, www.acadiau.ca, or www.msvu.ca?
Comment 3 Simon L. B. Nielsen freebsd_committer freebsd_triage 2012-12-22 16:17:03 UTC 
On 19 December 2012 16:53, Joseph Mingrone <jrm@ftfl.ca> wrote:
> On Mon, Dec 17, 2012 at 5:55 PM, Simon L. B. Nielsen <simon@freebsd.org> wrote:
>> Connecting to www.dal.ca port 80 with an IP from our delegated /24 fails with:
>>
>> # tcpdump -s0 -i bge0 host www.dal.ca
>> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
>> listening on bge0, link-type EN10MB (Ethernet), capture size 65535 bytes
>> 21:49:29.941767 IP routerer.FreeBSD.org.41224 > kil-cq5-4w0.ITS.Dal.Ca.http: Flags [S], seq 997436128, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 270875695 ecr 0], length 0
>> 21:49:32.940893 IP routerer.FreeBSD.org.41224 > kil-cq5-4w0.ITS.Dal.Ca.http: Flags [S], seq 997436128, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 270878695 ecr 0], length 0
>> 21:49:36.140885 IP routerer.FreeBSD.org.41224 > kil-cq5-4w0.ITS.Dal.Ca.http: Flags [S], seq 997436128, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 270881895 ecr 0], length 0
>> 21:49:39.340885 IP routerer.FreeBSD.org.41224 > kil-cq5-4w0.ITS.Dal.Ca.http: Flags [S], seq 997436128, win 65535, options [mss 1460], length 0
>> 21:49:42.540895 IP routerer.FreeBSD.org.41224 > kil-cq5-4w0.ITS.Dal.Ca.http: Flags [S], seq 997436128, win 65535, options [mss 1460], length 0
>> 21:49:45.740890 IP routerer.FreeBSD.org.41224 > kil-cq5-4w0.ITS.Dal.Ca.http: Flags [S], seq 997436128, win 65535, options [mss 1460], length 0
>> 21:49:51.940891 IP routerer.FreeBSD.org.41224 > kil-cq5-4w0.ITS.Dal.Ca.http: Flags [S], seq 997436128, win 65535, options [mss 1460], length 0
>>
>
> Are you able to connect any one of www.smu.ca, www.acadiau.ca, or www.msvu.ca?

Not from 8.8.178.0/24 - all requests time out.

-- 
Simon L. B. Nielsen
Comment 4 peter 2012-12-22 16:30:28 UTC 
On Sat, Dec 22, 2012 at 8:17 AM, Simon L. B. Nielsen <simon@freebsd.org> wrote:
> On 19 December 2012 16:53, Joseph Mingrone <jrm@ftfl.ca> wrote:
>> On Mon, Dec 17, 2012 at 5:55 PM, Simon L. B. Nielsen <simon@freebsd.org> wrote:
>>> Connecting to www.dal.ca port 80 with an IP from our delegated /24 fails with:
>>>
>>> # tcpdump -s0 -i bge0 host www.dal.ca
>>> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
>>> listening on bge0, link-type EN10MB (Ethernet), capture size 65535 bytes
>>> 21:49:29.941767 IP routerer.FreeBSD.org.41224 > kil-cq5-4w0.ITS.Dal.Ca.http: Flags [S], seq 997436128, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 270875695 ecr 0], length 0
>>> 21:49:32.940893 IP routerer.FreeBSD.org.41224 > kil-cq5-4w0.ITS.Dal.Ca.http: Flags [S], seq 997436128, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 270878695 ecr 0], length 0
>>> 21:49:36.140885 IP routerer.FreeBSD.org.41224 > kil-cq5-4w0.ITS.Dal.Ca.http: Flags [S], seq 997436128, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 270881895 ecr 0], length 0
>>> 21:49:39.340885 IP routerer.FreeBSD.org.41224 > kil-cq5-4w0.ITS.Dal.Ca.http: Flags [S], seq 997436128, win 65535, options [mss 1460], length 0
>>> 21:49:42.540895 IP routerer.FreeBSD.org.41224 > kil-cq5-4w0.ITS.Dal.Ca.http: Flags [S], seq 997436128, win 65535, options [mss 1460], length 0
>>> 21:49:45.740890 IP routerer.FreeBSD.org.41224 > kil-cq5-4w0.ITS.Dal.Ca.http: Flags [S], seq 997436128, win 65535, options [mss 1460], length 0
>>> 21:49:51.940891 IP routerer.FreeBSD.org.41224 > kil-cq5-4w0.ITS.Dal.Ca.http: Flags [S], seq 997436128, win 65535, options [mss 1460], length 0
>>>
>>
>> Are you able to connect any one of www.smu.ca, www.acadiau.ca, or www.msvu.ca?
>
> Not from 8.8.178.0/24 - all requests time out.

Hmm. We had a similar report about IPv6 reachability from the same
general area in Canada recently too.  I'll poke around a little bit
and see if I can see something.  Otherwise I'll file a ticket with the
netops folks to have them look at the bgp routing announcements.

-- 
Peter Wemm - peter@wemm.org; peter@FreeBSD.org; peter@yahoo-inc.com; KI6FJV
"All of this is for nothing if we don't go to the stars" - JMS/B5
"If Java had true garbage collection, most programs would delete
themselves upon execution." -- Robert Sewell
Comment 5 Joseph Mingrone freebsd_committer freebsd_triage 2013-01-08 15:15:04 UTC 
On Sat, Dec 22, 2012 at 12:30 PM, Peter Wemm <peter@wemm.org> wrote:
> Hmm. We had a similar report about IPv6 reachability from the same
> general area in Canada recently too.  I'll poke around a little bit
> and see if I can see something.  Otherwise I'll file a ticket with the
> netops folks to have them look at the bgp routing announcements.
>

That was the network administrator I contacted here at Dalhousie
University, although he doesn't seem convinced it's an IPv6 issue?

Has there been any progress resolving this issue?  I'm sorry to nag
(especially if it's not a problem on the FreeBSD end), but the research in
our group is quite reliant on FreeBSD (see
http://awarnach.mathstat.dal.ca/dokuwiki/doku.php?id=resources:home )
and we are unable to reach things like the source code repository,
mailing lists and the bug reports (all links from the mirrors (e.g.
www5) seem to point back to the main site).

Joseph
Comment 6 Simon L. B. Nielsen freebsd_committer freebsd_triage 2013-01-08 15:19:45 UTC 
On 8 January 2013 15:15, Joseph Mingrone <jrm@ftfl.ca> wrote:
> On Sat, Dec 22, 2012 at 12:30 PM, Peter Wemm <peter@wemm.org> wrote:
>> Hmm. We had a similar report about IPv6 reachability from the same
>> general area in Canada recently too.  I'll poke around a little bit
>> and see if I can see something.  Otherwise I'll file a ticket with the
>> netops folks to have them look at the bgp routing announcements.
>>
>
> That was the network administrator I contacted here at Dalhousie
> University, although he doesn't seem convinced it's an IPv6 issue?
>
> Has there been any progress resolving this issue?  I'm sorry to nag
> (especially if it's not a problem on the FreeBSD end), but the research in
> our group is quite reliant on FreeBSD (see
> http://awarnach.mathstat.dal.ca/dokuwiki/doku.php?id=resources:home )
> and we are unable to reach things like the source code repository,
> mailing lists and the bug reports (all links from the mirrors (e.g.
> www5) seem to point back to the main site).

I can't personally really do anything more with the network part, but
for the source code part you can get that from elsewhere, e.g.
svn0.us-east.freebsd.org. Mailing lists in theory has other non
FreeBSD.org mirrors, but they are less useful. For GNATS bug reports
you are probably out of luck currently.

-- 
Simon L. B. Nielsen
Comment 7 Joseph Mingrone freebsd_committer freebsd_triage 2013-01-22 18:55:01 UTC 
This PR can be closed now.  The network administrators here at
Dalhousie made some routing changes and everything seems to be working
now.
Comment 8 Mark Linimon freebsd_committer freebsd_triage 2013-01-23 03:05:16 UTC 
State Changed
From-To: open->closed

Closed at submitter's request.