Bug 242137
| Summary: | Dell R440 loses time on FreeBSD-12-STABLE following r352517, exceeding the NTP threshold of 500 PPM to correct the error. | ||
|---|---|---|---|
| Product: | Base System | Reporter: | Vinícius Zavam <egypcio> |
| Component: | bin | Assignee: | Cy Schubert <cy> |
| Status: | Closed Overcome By Events | ||
| Severity: | Affects Some People | CC: | alc, cy, stenn |
| Priority: | --- | ||
| Version: | 12.0-STABLE | ||
| Hardware: | amd64 | ||
| OS: | Any | ||
| Attachments: | |||
% ntpq -4 -n -c peers ntp1
remote refid st t when poll reach delay offset jitter
==============================================================================
-81.173.193.222 130.149.17.21 2 u 560 1024 377 1.318 1.931 0.427
-81.173.193.223 193.67.79.202 2 u 501 1024 377 1.325 1.580 0.148
*134.130.4.17 .GPS. 1 u 349 1024 377 2.087 -0.340 0.078
+134.130.5.17 .GPS. 1 u 679 1024 377 2.122 -0.343 0.090
+129.143.2.23 .GPS. 1 u 930 1024 377 7.557 -0.407 0.119
-129.143.2.33 .PZF. 1 u 446 1024 377 7.556 -0.612 0.055
-194.109.6.2 193.67.79.202 2 u 544 1024 377 5.820 -0.565 0.369
-213.136.0.252 .PPS. 1 u 659 1024 377 6.473 -0.132 0.958
-192.53.103.103 .PTB. 1 u 383 1024 377 13.873 1.432 1.090
-10.0.5.4 192.87.106.2 2 s 764 1024 376 0.218 -0.158 0.105
% ntpq -4 -n -c peers ntp2
remote refid st t when poll reach delay offset jitter
==============================================================================
-81.173.193.222 130.149.17.21 2 u 752 1024 377 1.197 2.366 0.228
-81.173.193.223 193.67.79.202 2 u 802 1024 377 1.457 1.595 0.211
+134.130.4.17 .GPS. 1 u 605 1024 377 2.156 -0.173 0.048
+134.130.5.17 .GPS. 1 u 966 1024 377 2.186 -0.194 0.032
+129.143.2.23 .GPS. 1 u 118 1024 377 7.666 -0.239 0.080
-129.143.2.33 .PZF. 1 u 503 1024 377 7.487 -0.448 0.056
-194.109.6.2 193.67.79.202 2 u 393 1024 377 5.515 -0.296 0.063
*192.87.106.2 .GPS. 1 u 601 1024 377 5.719 -0.084 0.091
+192.87.36.4 .MRS. 1 u 128 1024 377 8.280 0.013 0.084
-10.0.5.3 134.130.4.17 2 s 245 1024 377 0.202 0.150 0.092
*** same hardware running releng/12.1 works fine as well % ntpd --version ntpd 4.2.8p12-a (1) *** 12.1-STABLE amd64 r354893 1201502 % uptime 11:38AM up 1 day, 18:23, 5 users, load averages: 0.53, 0.34, 0.27 % cat /var/db/ntp/ntpd.drift 48.229 recently booted the machine using 'FreeBSD-13.0-CURRENT-amd64-20191121-r354935-memstick.img' and it behaves the same as stable/12@r354893. https://svnweb.freebsd.org/base/head/contrib/ntp/ntpd/?view=log suggests that 'ntpd' should be version 4.2.8p13, it's still 4.2.8p12 (1). just like under stable/12 (12.1-STABLE). how to reproduce? 1. boot usb stick with FreeBSD-13.0-CURRENT-amd64-20191121-r354935; 2. enter shell mode; 3. get your network settings done; 4. run `service ntpd onestart` and wait. FreeBSD freebsd-HEAD-13-0 13.0-CURRENT FreeBSD 13.0-CURRENT #0 r354935: Thu Nov 21 08:28:22 UTC 2019 root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64 11:33AM up 1 hr, 2 users, load averages: 0.14, 0.17, 0.21 ntpq> lpeers remote refid st t when poll reach delay offset jitter ============================================================================== 0.freebsd.pool. .POOL. 16 p - 64 0 0.000 0.000 0.000 +193.1.12.167 230.0.204.150 2 u 59 64 1 26.073 492.557 266.495 +82-64-45-50.sub .GPS. 1 u 58 64 1 32.215 493.964 265.248 +216.6.2.70 (up2 195.219.14.21 2 u 56 64 1 91.754 490.716 265.393 +server.samoylyk 161.143.24.141 2 u 52 64 1 24.803 500.755 299.806 +v35005.php-frie 131.188.3.223 2 u 40 64 1 4.615 528.354 314.264 +stratum2-3.NTP. 129.70.130.70 2 u 20 64 7 11.671 566.456 419.030 +time02.fal.code 192.53.103.108 2 u 18 64 7 9.590 570.200 421.709 +162.159.200.123 10.48.8.4 3 u 16 64 7 1.900 573.828 420.288 +213.251.53.187 195.66.241.10 2 u 16 64 7 5.425 573.531 422.477 +ntp1.rrze.uni-e .DCFp. 1 u 16 64 7 7.907 572.864 426.169 +x8d1ee404.agdsn 85.220.190.246 3 u 16 64 7 19.868 571.959 421.706 *ntp2.rrze.uni-e .MBGh. 1 u 19 64 37 7.964 567.335 384.663 ntpq> sysinfo associd=0 status=0613 leap_none, sync_ntp, 1 event, spike_detect, system peer: ntp2.rrze.uni-erlangen.de:123 system peer mode: client leap indicator: 00 stratum: 2 log2 precision: -23 root delay: 7.964 root dispersion: 617.125 reference ID: 131.188.3.222 reference time: e18245a6.3f00f24a Fri, Nov 22 2019 11:31:50.246 system jitter: 42.259079 clock jitter: 33.033 clock wander: 0.000 broadcast delay: -50.000 symm. auth. delay: 0.000 ntpq> version ntpq 4.2.8p12-a (1) that's odd: running the same 13.0-CURRENT r354935 live stick in another machine (regular PC, with an Intel Core i3-3110M@2.4Ghz) it works! the CPU on the server is an Intel Xeon Bronze 3204@1.9Ghz current status: till 12.1-RELEASE the box with the Xeon works fine, running stable/12 (r354367 and r354893) or HEAD (r354935) NTP does not work. Created attachment 209404 [details]
13-0-amd64-r354935_dmesg.txt
13-0-amd64-r354935_dmesg.txt
Created attachment 209405 [details]
13-0-amd64-r354935_pciconf.txt
13-0-amd64-r354935_pciconf.txt
root@13-0-amd64-r354935:~ # date
Mon Nov 25 09:33:27 UTC 2019
root@13-0-amd64-r354935:~ # uptime
9:33AM up 18 mins, 2 users, load averages: 0.05, 0.14, 0.10
ntpq> version
ntpq 4.2.8p12-a (1)
ntpq> sysinfo
associd=0 status=0613 leap_none, sync_ntp, 1 event, spike_detect,
system peer: zarkon.mcl.gg:123
system peer mode: client
leap indicator: 00
stratum: 3
log2 precision: -23
root delay: 42.766
root dispersion: 756.559
reference ID: 94.16.113.67
reference time: e1861e40.639d7150 Mon, Nov 25 2019 9:32:48.389
system jitter: 110.350937
clock jitter: 26.918
clock wander: 0.000
broadcast delay: -50.000
symm. auth. delay: 0.000
ntpq> lpeers
remote refid st t when poll reach delay offset jitter
==============================================================================
0.freebsd.pool. .POOL. 16 p - 64 0 0.000 0.000 0.000
+frank.askja.de 124.216.164.14 2 u 115 64 1 2.378 549.955 329.626
+spacys.de 130.149.17.8 2 u 114 64 1 26.613 542.582 321.933
+185.90.160.100 100.10.69.89 2 u 113 64 1 5.412 553.136 328.887
+meinekiste.de 170.231.250.158 2 u 115 64 1 9.228 550.012 330.670
+245.ip-51-75-74 82.161.139.11 2 u 26 64 17 17.817 713.543 514.645
+server.spnr.de 124.216.164.14 2 u 47 64 73 25.461 667.287 492.443
*zarkon.mcl.gg 131.188.3.222 2 u 53 64 77 35.762 650.561 439.307
+hotel.zq1.de 161.62.157.173 3 u 46 64 77 30.966 666.341 453.826
using pkg's ntp package didn't workaround the issue; ntp-4.2.8p13_5 openntpd-6.2p3_4,2
% ntpctl -s all
4/4 peers valid, constraint offset 0s, clock unsynced, clock offset is 536.865ms
peer
wt tl st next poll offset delay jitter
62.116.130.3 from pool pool.ntp.org
1 10 2 23s 30s 419.828ms 24.961ms 17.435ms
131.188.3.221 from pool pool.ntp.org
1 10 1 11s 33s 386.589ms 25.516ms 17.142ms
188.68.36.203 from pool pool.ntp.org
1 10 2 20s 30s 406.697ms 24.534ms 20.615ms
185.244.195.159 from pool pool.ntp.org
1 10 2 24s 33s 410.479ms 23.648ms 17.262ms
chrony-3.5 chronyc> tracking Reference ID : 8AC95ABD (ns1.your-ns.de) Stratum : 3 Ref time (UTC) : Mon Nov 25 17:48:09 2019 System time : 0.976312995 seconds slow of NTP time Last offset : -0.117800929 seconds RMS offset : 0.117547080 seconds Frequency : 0.000 ppm slow Residual freq : +198.180 ppm Skew : 1000000.000 ppm Root delay : 0.064190254 seconds Root dispersion : 64.806434631 seconds Update interval : 64.7 seconds Leap status : Normal Read this carefully. There are a number of outputs being requested here. Your network or upstream has issues. Compare offset and jitter of the server that has the error with the server that does not. You've only been able to reach once (or 7 times). Compere this with those that work. Can we have output of cat /var/db/ntp/ntp.drift please? Large drift in conjunction with the inability to contact each ntp sever is the cause of this error. You should see synchronization flags (*, #, +, -) in your ntpq output. We see none. You're not synchronizing. Again, this points to probable problems with either your local network, upstream, or denied by your firewall. ---- Try adding rlimit memlock 32 to ntp.conf. (This will not work with ASLR enabled.) This was the only change to ntp prior to r354367. (With ASLR enabled, mlock() caused a segfault and stack corruption.) How much RAM is installed on this machaine? How much swap is configured? How much swap is in use? ---- Please answer all of the questions above. Again: 1. Can you ping your ntp servers? What is the round trip time? 2. Is UDP/123 open? 3. What is in ntp.drift? 4. How much RAM is in the machine? 5. Is swap being used and is ntpd using some of that swap? 6. Does setting memlock in ntp.conf do anything for you? (In reply to Cy Schubert from comment #15) hey, thanks for sharing some thoughts here cy@. appreciated! I'm curious to check and understand how you meant that it's a network issue; as other machines running in the same network sync their clock waaaaay faster and with no issues like this one reported here. would really like to dig more into it. the ntp.drift was posted on "Comment 5": its value was 48.229 after 1d18h of uptime with ntpd running. "Comment 1" has the output of two of the ntp servers that I use to synchonize this ntpd instance. AFAIK, both are running fine - ntp1 is on the same network as the problematic machine. at the end of the initial report you can see the flags, just like on Comments 7 and 11 - they are from the same machine (once running stable/12, later running head). so, as I do have a very identical box to use it was moved to a different location and I cloned the system from the original server and had the same issue in another location (free of firewalling though); when I boot 12.1-RELEASE it just works as expected. ----- 1. yes; min/avg/max/mdev = 0.119/0.216/0.284/0.057 ms 2. yes; 3. 48.229 (posted before); 4. 32G; 5. no; 6. did not change a thing, used default (FreeBSD's) ntp.conf during tests only. I assume you restarted ntpd after setting rlimit memlock 32 ? 1. Stop ntpd.
2. tcpdump -i YOUR_EXTERNAL_INTERFACE udp port 123
or if you prefer not to post here
2. tcpdump -i YOUR_EXTERNAL_INTERFACE -w ntp.pcap udp port 123
and email the output file to my freebsd.org email address (I'll review using wireshark)
3. start ntpd
4. Then either post the output or email the file to me. I suspect ntpd is sending but not receiving.
(In reply to Cy Schubert from comment #17) I did not test it. I used only default ntp.conf shipped with FreeBSD without any changes. but yes, if I change the ntp.conf I do restart its daemon right after :) forgot to mention one thing though: what I did try was changing 'kern.timecounter.hardware' to other values instead os just TSC. I am working here to get a side-by-side scenario with 2 machines before starting the tcpdump - still curious why: the machine works fine when I run releng/12.1 as I mentioned before thanks! I'm running 'tcpdump' and playing with 2 different machines, both running HEAD/amd64 r354935. one is the R440 (bad guy) and the other one is just a regular PC.
the other R440 I left running with stable/12 r354893 for a couple of days has these information to share:
% uptime
11:50AM up 5 days, 18:22, 1 user, load averages: 0.27, 0.24, 0.24
% cat /var/db/ntp/ntpd.drift
80.938
% ntpq -4 -n -p
remote refid st t when poll reach delay offset jitter
==============================================================================
10.0.5.3 134.130.4.17 2 u 59 64 1 0.338 311.293 0.000
10.0.5.4 192.87.106.2 2 u 8 64 1 0.182 420.868 78.854
10.0.32.248 134.130.4.17 2 u 60 64 1 1.105 307.941 0.000
10.0.32.249 192.87.106.2 2 u 10 64 1 0.980 416.475 77.317
here we do not see the flags, but if I try to get the peers list from ***other servers*** (also running FreeBSD) in the very same network and using the very same ntp.conf, we have the following:
12.1-RELEASE-p1% ntpq -4 -n -p
remote refid st t when poll reach delay offset jitter
==============================================================================
*10.0.5.3 134.130.4.17 2 u 121 1024 377 0.650 0.747 0.460
+10.0.5.4 192.87.106.2 2 u 627 1024 377 0.560 -0.909 0.834
-10.0.32.248 134.130.4.17 2 u 37 1024 377 2.279 -0.666 3.731
+10.0.32.249 192.87.106.2 2 u 487 1024 377 1.602 -0.848 0.955
12.1-STABLE-r352727% ntpq -4 -n -p
remote refid st t when poll reach delay offset jitter
==============================================================================
*10.0.5.3 134.130.4.17 2 u 99 128 377 0.081 0.315 0.232
+10.0.5.4 192.87.106.2 2 u 111 128 377 0.150 -1.349 0.132
+10.0.32.248 134.130.4.17 2 u 5 128 377 0.889 -0.554 7.056
-10.0.32.249 192.87.106.2 2 u 12 128 377 0.888 -1.363 1.518
Created attachment 209450 [details]
STABLE_12_r354893_ntpd.log
attaching the ntpd.log from yesterday, which is running for about 5 days.
booted an USB memstick and let it run for a while, using all the defaults we have shipped with the live/install image:
root@HEADamd64r354935-a:~ # uptime
12:08PM up 3:18, 2 users, load averages: 0.17, 0.25, 0.24
root@HEADamd64r354935-a:~ # grep \$FreeBSD /etc/ntp.conf
# $FreeBSD: head/usr.sbin/ntp/ntpd/ntp.conf 352304 2019-09-13 20:20:05Z cy $
root@HEADamd64r354935-a:~ # grep -v -E '^#|^$' /etc/ntp.conf
tos minclock 3 maxclock 6
pool 0.freebsd.pool.ntp.org iburst
restrict default limited kod nomodify notrap noquery nopeer
restrict source limited kod nomodify notrap noquery
restrict 127.0.0.1
restrict ::1
leapfile "/var/db/ntpd.leap-seconds.list"
root@HEADamd64r354935-a:~ # cat /var/db/ntp/ntpd.drift
500.000
root@HEADamd64r354935-a:~ # ntpq -4 -n -p
remote refid st t when poll reach delay offset jitter
==============================================================================
0.freebsd.pool. .POOL. 16 p - 64 0 0.000 0.000 0.000
+116.203.209.35 237.17.204.95 2 u 39 64 3 7.661 844.631 295.042
+80.67.16.8 131.188.3.221 2 u 38 64 3 1.761 1420.93 511.244
+144.76.70.233 152.103.15.66 2 u 37 64 3 9.098 1189.64 313.973
+193.30.120.245 192.53.103.108 2 u 36 64 3 7.786 847.167 298.156
+203.107.6.88 10.137.38.86 2 u 22 64 3 364.706 1125.61 289.341
+195.46.37.22 194.146.251.101 2 u 7 64 3 33.843 904.179 301.022
*213.136.0.252 .PPS. 1 u 9 64 3 6.729 1011.97 254.233
+163.237.218.19 .GPS. 1 u 5 64 3 98.093 1249.28 299.505
+188.125.64.6 212.82.106.33 2 u 3 64 3 23.957 1252.51 329.107
+85.195.227.162 124.216.164.14 2 u 66 64 3 9.857 795.025 292.792
+185.17.70.106 162.23.41.55 2 u 5 64 3 14.402 1136.51 264.795
+193.140.100.40 160.45.10.8 2 u 4 64 3 62.183 1136.77 266.951
root@HEADamd64r354935-a:~ # nc -nzvu 85.195.227.162 123
Connection to 85.195.227.162 123 port [udp/*] succeeded!
root@HEADamd64r354935-a:~ # nc -nzvu 163.237.218.19 123
Connection to 163.237.218.19 123 port [udp/*] succeeded!
root@HEADamd64r354935-a:~ # nc -nzvu 131.188.3.221 123
Connection to 131.188.3.221 123 port [udp/*] succeeded!
now the other machine that works (same version and rev from machine A, and same ntp.conf):
root@HEADamd64r354935-b:~ # uptime
12:12PM up 3:27, 2 users, load averages: 0.12, 0.14, 0.09
root@HEADamd64r354935-b:~ # cat /var/db/ntp/ntpd.drift
11.226
root@HEADamd64r354935-b:~ # ntpq -4 -n -p
remote refid st t when poll reach delay offset jitter
==============================================================================
0.freebsd.pool. .POOL. 16 p - 64 0 0.000 0.000 0.000
+145.239.0.227 131.188.3.223 2 u 46 64 377 17.364 -14.710 1.288
+217.144.138.234 131.188.3.220 2 u 23 64 377 11.204 -14.875 2.140
-88.99.216.194 131.188.3.223 2 u 6 64 377 16.726 -14.850 5.395
-176.9.1.211 161.62.157.173 3 u 100 64 376 15.619 -16.407 7.332
*62.128.1.18 213.95.151.123 2 u 24 64 377 11.255 -15.058 6.604
root@HEADamd64r354935-b:~ # nc -nzvu 88.99.216.194 123
Connection to 88.99.216.194 123 port [udp/*] succeeded!
root@HEADamd64r354935-b:~ # nc -nzvu 217.144.138.234 123
Connection to 217.144.138.234 123 port [udp/*] succeeded!
netcat from my workstation to both machines (first machine A, than machine B): % nc -nzvu 10.0.86.163 123 Connection to 10.0.86.163 123 port [udp/*] succeeded! % nc -nzvu 10.0.128.203 123 Connection to 10.0.128.203 123 port [udp/*] succeeded! PS: mailed you the tcpdump from both machines Looking at tcpdump outputs before I go to $JOB, I'll put this here so you can check this out: 10.0.128.203 and 10.0.86.163 are serving up garbage. The NTP packets say they are stratum 0 time servers (unspecified or invalid) and their reference, origin, and receive timestamps are Feb 7, 2036 (the fields are 0's). What are the A and B servers IP's again? (In reply to Cy Schubert from comment #25) 10.0.86.163 == a 10.0.128.203 == b both, A and B, are using 'pool 0.freebsd.pool.ntp.org' and are not meant to really work as NTP servers, just use ntp as client - like the default ntp.conf we have right now (Comment 22). the actual NTP servers I have are all working just fine though. That doesn't explain why you are being served up bad time by the servers in your 10.0 network. That's your problem. Fix that and your problem is fixed. 0.freebsd.pool.ntp.org does not contain 10.0 servers in its definition. YOur ntp.conf from reply #1 does not include 0.freebsd.pool.ntp.org: restrict default limited kod nomodify notrap noquery nopeer restrict 127.0.0.1 mask 255.255.255.255 restrict 10.64.35.0 mask 255.255.255.248 nomodify notrap server ntp1 iburst prefer server ntp2 iburst prefer server ntp3 iburst server ntp4 iburst tos orphan 3 statistics clockstats cryptostats loopstats peerstats rawstats sysstats in total, I have 3 machines: 2 are R440 (which are impacted with this issues to synchronize time) and 1 is just an offset box. machine X is 10.0.5.7 (running since 5 days, with NTP sync issues); machine A is 10.0.86.163 (very same hw as machine X, live img running); machine B is 10.0.128.203 (offset box, live img running). ntp servers 1, 2, 3, and 4 are fine: 2 of them are on 10.0.5.x, and 2 are on 10.0.248.x. no issues at all. these are the current internal stratum 2 servers. machines X and A can't work as expected and use ntpd to synchronize their clocks; as we see here the drifts are kinda high, offset and jitter are pretty bad although the delay/latency to reach internal and external NTP servers is not a problem. hope that gives all a better picture of the scenario here. maybe I should have written it before. (In reply to Cy Schubert from comment #28) no no no. this config you pasted is definitely *NOT* used to sync against pool.ntp.org! the actual ntp.conf for internal use is: ----- restrict default limited kod nomodify notrap noquery nopeer restrict 127.0.0.1 mask 255.255.255.255 restrict 10.0.0.0 mask 255.255.0.0 nomodify notrap restrict 10.64.0.0 mask 255.255.0.0 nomodify notrap server ntp1 iburst prefer server ntp2 iburst prefer server ntp3 iburst # prefer server ntp4 iburst # prefer tos orphan 3 ----- the 'statistics' was added just to collect data; the restrict was modified before but THIS ONE is the one used in prod for this environment current test setup now on machine A and B: both bare metal with live images running HEAD amd64 r354935. # ntp.conf tos minclock 3 maxclock 6 pool 0.freebsd.pool.ntp.org iburst restrict default limited kod nomodify notrap noquery nopeer restrict source limited kod nomodify notrap noquery restrict 127.0.0.1 restrict ::1 leapfile "/var/db/ntpd.leap-seconds.list" after getting network to work, just ran # service ntpd onestart short update on the 'ntpq -p' for machine A and B (with their flags):
----------
root@HEADamd64r354935-a:~ # uptime
2:45PM up 5:55, 2 users, load averages: 0.17, 0.28, 0.27
root@HEADamd64r354935-a:~ # ntpq -4 -n -p
remote refid st t when poll reach delay offset jitter
==============================================================================
0.freebsd.pool. .POOL. 16 p - 64 0 0.000 0.000 0.000
#82.100.248.10 194.97.129.17 2 u 54 64 1 4.631 528.815 222.540
+131.234.137.63 .DCF. 1 u 50 64 1 13.021 881.310 243.876
#138.201.16.225 130.149.17.21 2 u 50 64 1 9.245 532.426 226.107
#5.189.146.13 131.188.3.220 2 u 50 64 1 14.836 655.488 141.082
#176.9.166.35 192.53.103.108 2 u 34 64 1 9.306 685.282 147.836
#193.30.120.245 192.53.103.108 2 u 16 64 1 7.779 943.407 268.613
+173.249.58.145 131.188.3.222 2 u 12 64 1 15.856 951.403 270.882
+37.120.184.82 131.188.3.222 2 u 56 64 1 7.635 869.279 239.843
+188.68.36.203 131.188.3.221 2 u 57 64 1 7.623 752.591 142.400
+80.67.16.8 3.121.254.221 3 u 58 64 1 1.762 754.617 142.360
+213.239.239.166 237.17.204.95 2 u 57 64 1 9.279 756.224 146.244
*5.199.135.170 130.149.17.21 2 u 56 64 1 2.139 868.705 239.148
root@HEADamd64r354935-a:~ # cat /var/db/ntp/ntpd.drift
500.000
----------
root@HEADamd64r354935-b:~ # uptime
2:46PM up 6:01, 2 users, load averages: 0.10, 0.14, 0.09
root@HEADamd64r354935-b:~ # ntpq -4 -n -p
remote refid st t when poll reach delay offset jitter
==============================================================================
0.freebsd.pool. .POOL. 16 p - 64 0 0.000 0.000 0.000
+178.162.199.162 131.188.3.223 2 u 15 128 377 20.890 -9.873 26.352
+144.76.0.164 193.67.79.202 2 u 1 128 377 16.949 -9.525 2.543
*90.187.99.165 .PPS. 1 u 123 128 377 32.826 -12.698 2.729
+217.91.44.17 192.168.100.15 2 u 117 128 377 23.217 -10.839 3.000
+80.153.195.191 192.168.100.9 2 u 124 128 377 25.459 -7.415 2.383
root@HEADamd64r354935-b:~ # cat /var/db/ntp/ntpd.drift
6.134
regarding the comments about "network issues": how would that explain a pretty normal behavior of NTP running 12.1-RELEASE on the machine A (on the very same network)? After briefly looking at the pcap files you sent me, it's not a network issue but the ntp servers on your 10.0 network are serving bad time. They say they are stratum 0 time servers, which is invalid and the three time fields in the ntp packets are zero. The ntpq -p from your A and B servers look good. (In reply to Cy Schubert from comment #34) % host ntp ntp.localdomain has address 10.0.5.4 ntp.localdomain.com has address 10.0.32.249 ntp.localdomain.com has address 10.0.32.248 ntp.localdomain.com has address 10.0.5.3 % ntpdate -q ntp server 10.0.5.4, stratum 2, offset -0.000039, delay 0.02678 server 10.0.32.249, stratum 2, offset 0.000023, delay 0.02589 server 10.0.32.248, stratum 2, offset -0.000080, delay 0.02583 server 10.0.5.3, stratum 2, offset -0.000569, delay 0.02702 27 Nov 09:43:16 ntpdate[11451]: adjust time server 10.0.32.248 offset -0.000080 sec ----- % ntpq -4 -n -p ntp1 remote refid st t when poll reach delay offset jitter ============================================================================== -81.173.193.222 131.188.3.220 2 u 979 1024 377 1.343 2.425 0.145 -81.173.193.223 193.67.79.202 2 u 736 1024 377 1.176 1.530 0.307 *134.130.4.17 .GPS. 1 u 890 1024 377 2.168 0.380 0.305 +134.130.5.17 .GPS. 1 u 808 1024 377 2.055 0.293 0.369 +129.143.2.23 .GPS. 1 u 352 1024 377 7.571 0.311 0.302 +129.143.2.33 .PZF. 1 u 811 1024 377 7.481 0.196 0.192 -194.109.6.2 193.79.237.14 2 u 293 1024 377 5.773 0.022 0.277 +213.136.0.252 .PPS. 1 u 962 1024 377 6.243 0.202 0.511 +192.53.103.103 .PTB. 1 u 927 1024 377 10.518 0.404 1.809 -10.0.5.4 192.87.106.2 2 s 349 1024 337 0.183 0.729 0.162 % ntpq -4 -n -p ntp2 remote refid st t when poll reach delay offset jitter ============================================================================== -81.173.193.222 130.149.17.21 2 u 416 1024 377 1.172 1.517 0.120 -81.173.193.223 193.67.79.202 2 u 726 1024 377 1.031 0.762 0.294 +134.130.4.17 .GPS. 1 u 255 1024 377 2.217 -0.387 0.176 +134.130.5.17 .GPS. 1 u 718 1024 377 2.150 -0.310 0.104 -129.143.2.23 .GPS. 1 u 149 1024 377 7.572 -0.480 0.180 -129.143.2.33 .PZF. 1 u 852 1024 377 7.477 -0.650 0.218 -194.109.6.2 193.67.79.202 2 u 568 1024 377 5.456 -0.565 0.046 *192.87.106.2 .GPS. 1 u 542 1024 377 5.897 -0.054 0.100 +192.87.36.4 .MRS. 1 u 36 1024 377 8.288 -0.209 0.210 -10.0.5.3 134.130.4.17 2 s 367 1024 176 0.066 -0.670 0.205 % ntpq -4 -n -p ntp3 remote refid st t when poll reach delay offset jitter ============================================================================== -81.173.193.222 131.188.3.220 2 u 961 1024 377 1.851 1.497 0.454 -81.173.193.223 193.67.79.202 2 u 102 1024 377 1.499 0.702 2.060 *134.130.4.17 .GPS. 1 u 489 1024 377 2.481 -0.365 4.585 +134.130.5.17 .GPS. 1 u 912 1024 377 2.419 -0.361 0.426 +129.143.2.23 .GPS. 1 u 371 1024 377 7.941 -0.306 0.312 +129.143.2.33 .PZF. 1 u 994 1024 377 7.829 -0.547 0.330 -194.109.6.2 193.67.79.202 2 u 462 1024 377 6.535 -0.867 6.943 -213.136.0.252 .PPS. 1 u 394 1024 377 8.033 -0.973 5.084 +192.53.103.103 .PTB. 1 u 346 1024 377 10.727 -0.306 1.693 -10.0.32.249 192.87.106.2 2 s 580 1024 376 0.180 0.157 0.329 % ntpq -4 -n -p ntp4 remote refid st t when poll reach delay offset jitter ============================================================================== -81.173.193.222 131.188.3.220 2 u 781 1024 377 2.001 1.389 6.224 -81.173.193.223 193.67.79.202 2 u 277 1024 377 1.477 0.594 1.523 +134.130.4.17 .GPS. 1 u 511 1024 377 2.537 -0.467 8.575 +134.130.5.17 .GPS. 1 u 41 1024 377 2.467 -0.515 1.370 -129.143.2.23 .GPS. 1 u 577 1024 377 7.978 -0.511 0.063 -129.143.2.33 .PZF. 1 u 376 1024 377 7.928 -0.759 2.418 -194.109.6.2 193.67.79.202 2 u 812 1024 377 6.086 -0.628 0.657 *192.87.106.2 .GPS. 1 u 3 1024 377 6.066 -0.334 0.128 +192.87.36.4 .MRS. 1 u 125 1024 377 8.419 -0.356 0.106 -10.0.32.248 134.130.4.17 2 s 316 1024 377 0.141 -0.138 0.321 ----- % ntpq -4 -n -c sysinfo ntp1 associd=0 status=0615 leap_none, sync_ntp, 1 event, clock_sync, system peer: 134.130.4.17:123 system peer mode: client leap indicator: 00 stratum: 2 log2 precision: -24 root delay: 2.050 root dispersion: 19.509 reference ID: 134.130.4.17 reference time: e188c440.20a64f14 Wed, Nov 27 2019 9:45:36.127 system jitter: 0.495904 clock jitter: 0.264 clock wander: 0.055 broadcast delay: -50.000 symm. auth. delay: 0.010 % ntpq -4 -n -c sysinfo ntp2 associd=0 status=0615 leap_none, sync_ntp, 1 event, clock_sync, system peer: 192.87.106.2:123 system peer mode: client leap indicator: 00 stratum: 2 log2 precision: -24 root delay: 5.897 root dispersion: 45.148 reference ID: 192.87.106.2 reference time: e188bd76.faadd7b2 Wed, Nov 27 2019 9:16:38.979 system jitter: 0.100174 clock jitter: 0.138 clock wander: 0.013 broadcast delay: -50.000 symm. auth. delay: 0.008 % ntpq -4 -n -c sysinfo ntp3 associd=0 status=0615 leap_none, sync_ntp, 1 event, clock_sync, system peer: 134.130.4.17:123 system peer mode: client leap indicator: 00 stratum: 2 log2 precision: -24 root delay: 2.481 root dispersion: 34.979 reference ID: 134.130.4.17 reference time: e188c1b9.3abded05 Wed, Nov 27 2019 9:34:49.229 system jitter: 4.585486 clock jitter: 0.170 clock wander: 0.037 broadcast delay: -50.000 symm. auth. delay: 0.000 % ntpq -4 -n -c sysinfo ntp4 associd=0 status=0615 leap_none, sync_ntp, 1 event, clock_sync, system peer: 192.87.106.2:123 system peer mode: client leap indicator: 00 stratum: 2 log2 precision: -24 root delay: 6.066 root dispersion: 21.975 reference ID: 192.87.106.2 reference time: e188c3a1.1ddeb161 Wed, Nov 27 2019 9:42:57.116 system jitter: 0.128426 clock jitter: 0.153 clock wander: 0.014 broadcast delay: -50.000 symm. auth. delay: 0.000 (In reply to Vinícius Zavam from comment #35) DISCLAIMER: previous 'host' command returns % host ntp ntp.localdomain has address 10.0.5.4 ntp.localdomain has address 10.0.32.249 ntp.localdomain has address 10.0.32.248 ntp.localdomain has address 10.0.5.3 just replaced domain's name; let's focus on the NTP thing. DNS is definitely working! I mailed you the pcap file which I ran in my workstation, using these 4 servers. *** ran the "Platform Specific Bootable ISO, PER440,19.10.00" to update BIOS and other firmwares, but no updates were applied - the box is already with latest stuff. https://www.dell.com/support/home/us/en/04/drivers/driversdetails?driverid=1098x&oscode=naa&productcode=poweredge-r440 this is machine X and A ^ Need a reset here: Can you list the IP addresses of your 12.1, A and B servers, please. Also list ntp.conf for each. machine X; 10.0.5.7 (stable/12@r354893, 12.1-STABLE) machine A; 10.0.86.163 (head@r354935) machine B; 10.0.128.203 (head@r354935) # grep -v -E '^#|^$' /etc/ntp.conf tos minclock 3 maxclock 6 pool 0.freebsd.pool.ntp.org iburst restrict default limited kod nomodify notrap noquery nopeer restrict source limited kod nomodify notrap noquery restrict 127.0.0.1 restrict ::1 leapfile "/var/db/ntpd.leap-seconds.list" (In reply to Cy Schubert from comment #25) regarding the timestamp with 'Feb 7 2036' you mentioned before: I've tcpdumped other FreeBSD machines and they all have the same "issue". these recently tested FreeBSD machines are not even in the same location/country, and therefore can't be influenced by the other machines/servers mentioned before. they run the following revisions: r353654 e r354990 and r354992 (HEAD/amd64). they're all using the same default ntp.conf from FreeBSD, just like the one pasted on Comment 40. this is how the time stamping looks like: Network Time Protocol (NTP Version 4, client) Flags: 0xe3, Leap Indicator: unknown (clock unsynchronized), Version number: NTP Version 4, Mode: client 11.. .... = Leap Indicator: unknown (clock unsynchronized) (3) ..10 0... = Version number: NTP Version 4 (4) .... .011 = Mode: client (3) [Response In: 2] Peer Clock Stratum: unspecified or invalid (0) Peer Polling Interval: 6 (64 seconds) Peer Clock Precision: 0.000000 seconds Root Delay: 0.000000 seconds Root Dispersion: 0.000000 seconds Reference ID: (Initialization) Reference Timestamp: Feb 7, 2036 06:28:16.000000000 UTC Origin Timestamp: Feb 7, 2036 06:28:16.000000000 UTC Receive Timestamp: Feb 7, 2036 06:28:16.000000000 UTC Transmit Timestamp: Nov 29, 2019 14:45:11.501112827 UTC ----- Network Time Protocol (NTP Version 4, client) Flags: 0x23, Leap Indicator: no warning, Version number: NTP Version 4, Mode: client 00.. .... = Leap Indicator: no warning (0) ..10 0... = Version number: NTP Version 4 (4) .... .011 = Mode: client (3) [Response In: 58] Peer Clock Stratum: secondary reference (2) Peer Polling Interval: 6 (64 seconds) Peer Clock Precision: 0.000000 seconds Root Delay: 0.009277 seconds Root Dispersion: 0.940109 seconds Reference ID: 76.127.35.142 Reference Timestamp: Nov 29, 2019 14:45:18.493365262 UTC Origin Timestamp: Feb 7, 2036 06:28:16.000000000 UTC Receive Timestamp: Feb 7, 2036 06:28:16.000000000 UTC Transmit Timestamp: Nov 29, 2019 14:45:20.484546625 UTC Created attachment 209680 [details]
STABLE_12_r354367_dmesg.boot.txt
https://www.dell.com/en-us/work/shop/povw/poweredge-r440 + https://ark.intel.com/content/www/us/en/ark/products/193381/intel-xeon-bronze-3204-processor-8-25m-cache-1-90-ghz.html the 'rlimit memlock 32' option was not added to the ntp.conf on my side because it seems to be the default already.
from the ntp.conf(5) manpage:
memlock Nmegabytes
Specify the number of megabytes of memory that should be
allocated and locked. Probably only available under
Linux, this option may be useful when dropping root (the
-i option). The default is 32 megabytes on non-Linux
machines, and -1 under Linux. -1 means "do not lock the
process into memory". 0 means "lock whatever memory the
process wants into memory".
Created attachment 209689 [details]
STABLE_12_r354367_ntpq_rootdisp.png
12.1-STABLE, stable/12 [r354367] is the dark blue line.
used `ntpq -4 -n -c rl` to collect data from *all* these 5 machines.
(In reply to Vinícius Zavam from comment #44) rlimit memlock 32 is not the default. It was removed last summer because it caused ntp to segfault with ASLR and stack gap != 0. There was a discussion about this on -current at the time. Added Harlan Stenn of ntp.org to this discussion. (In reply to Cy Schubert from comment #46) oh, so... we could update the manpage; I might take a time to patch it later if we need to. thanks for reporting it - I missed that thread. I am going to add it to the .conf and restart the daemon. thanks! same. % ntpdate -4 -q -d 10.0.5.7 4 Dec 14:19:20 ntpdate[14809]: ntpdate 4.2.8p12-a (1) transmit(10.0.5.7) receive(10.0.5.7) 10.0.5.7: Server dropped: strata too high server 10.0.5.7, port 123 stratum 16, precision -22, leap 11, trust 000 refid [STEP], root delay 0.000000, root dispersion 0.000519 reference time: (no time) originate timestamp: e1923ee7.37e78eb9 Wed, Dec 4 2019 14:19:19.218 transmit timestamp: e1923ee8.24faa500 Wed, Dec 4 2019 14:19:20.144 delay 0.02658, dispersion 0.00000, offset -0.926634 4 Dec 14:19:20 ntpdate[14809]: no server suitable for synchronization found % ntpq -4 -n -c rl 10.0.5.7 associd=0 status=c018 leap_alarm, sync_unspec, 1 event, no_sys_peer, version="ntpd 4.2.8p12-a (1)", processor="amd64", system="FreeBSD/12.1-STABLE", leap=11, stratum=16, precision=-22, rootdelay=0.000, rootdisp=0.780, refid=STEP, reftime=(no time), clock=e1923ef9.0dc12b58 Wed, Dec 4 2019 14:19:37.053, peer=0, tc=6, mintc=3, offset=0.000000, frequency=128.357, sys_jitter=0.000238, clk_jitter=0.000, clk_wander=0.000, tai=37, leapsec=201701010000, expire=202006280000 booted the FreeBSD-12.1-STABLE-amd64-20191127-r355118-mini-memstick.img and performed `service ntpd onestart` (after getting a work network connection, of course) and was presented to the very same behavior :"( https://download.freebsd.org/ftp/snapshots/amd64/amd64/ISO-IMAGES/12.1/FreeBSD-12.1-STABLE-amd64-20191127-r355118-mini-memstick.img I am adding a few outputs here from truss and ktrace, which I ran in a FreeBSD HEAD@r355121 (amd64 bare metal hardware; dell poweredge r440, as reported before) https://share.riseup.net/#lzApqrgFltk9J4rlH11Zdg >>> freebsd-head-r355121-amd64_ktrace_output_SERVICE_ntpd_ONESTART.bin https://share.riseup.net/#LVlKzFFoHiylOMkfu58xng >>> ktrace_output_ntpq.bin https://share.riseup.net/#SQCa-pCEZmTi2wB4nX8O1Q >>> freebsd-head-r355121-amd64_truss_output_SERVICE_ntpd_ONESTART.txt https://share.riseup.net/#Dk9Kn_Wipb-aXNkN6l-BYA >>> truss_output_ntpq.txt % ntpq -4 -n -c rl -wp 127.0.0.1
associd=0 status=c613 leap_alarm, sync_ntp, 1 event, spike_detect,
version="ntpd 4.2.8p12-a (1)", processor="amd64",
system="FreeBSD/13.0-CURRENT", leap=11, stratum=3, precision=-23,
rootdelay=17.795, rootdisp=1311.617, refid=213.239.239.164,
reftime=e194ad60.0e89a5e8 Fri, Dec 6 2019 10:35:12.056,
clock=e194aeb2.99e66d49 Fri, Dec 6 2019 10:40:50.601, peer=14176, tc=6,
mintc=3, offset=0.000000, frequency=500.000, sys_jitter=173.114733,
clk_jitter=0.000, clk_wander=0.000, tai=37, leapsec=201701010000,
expire=201912280000
remote refid st t when poll reach delay offset jitter
==============================================================================
0.freebsd.pool.ntp.org
.POOL. 16 p - 64 0 0.000 0.000 0.000
+195.50.171.101 192.53.103.108 2 u 49 64 7 5.545 580.506 382.202
+35.158.196.249 192.53.103.108 2 u 51 64 7 4.671 576.939 382.105
#5.9.121.21 131.188.3.222 2 u 44 64 7 9.112 585.911 382.935
#78.46.241.151 213.239.239.165 3 u 49 64 7 9.312 454.438 479.780
+81.7.16.52 237.17.204.95 2 u 30 64 7 18.855 736.585 321.421
#94.16.114.254 5.9.121.21 3 u 21 64 7 7.765 869.343 305.652
#144.76.96.7 205.46.178.169 2 u 13 64 7 9.029 460.459 528.144
+193.141.27.6 192.53.103.108 2 u 14 64 7 6.719 872.974 303.831
#144.76.59.37 195.145.119.188 2 u 49 64 7 9.213 452.879 480.766
+159.69.150.81 237.17.204.95 2 u 46 64 7 7.735 809.909 292.674
+136.243.7.20 0.242.242.75 3 u 18 64 7 9.098 750.452 325.521
*213.239.239.164 205.46.178.169 2 u 16 64 17 7.723 1215.27 493.678
KTRACE: https://share.riseup.net/#0ZHiHkGHnc7rBcWTenvmeQ
TRUSS: https://share.riseup.net/#MtAe5XFYrG81DV_MCUHMOQ
root@freebsd-head-r355121-amd64:~ # grep -E -v '^$|^#' /etc/ntp.conf
tos minclock 3 maxclock 6
pool 0.freebsd.pool.ntp.org iburst
restrict default limited kod nomodify notrap noquery nopeer
restrict source limited kod nomodify notrap noquery
restrict 127.0.0.1
restrict ::1
leapfile "/var/db/ntpd.leap-seconds.list"
# ntpdate -4 -q -d 127.0.0.1 6 Dec 10:45:11 ntpdate[3328]: ntpdate 4.2.8p12-a (1) transmit(127.0.0.1) receive(127.0.0.1) 127.0.0.1: Server dropped: leap not in sync server 127.0.0.1, port 123 stratum 3, precision -23, leap 11, trust 000 refid [144.76.59.37], root delay 0.017899, root dispersion 2.040878 reference time: e194afb0.dc30c0aa Fri, Dec 6 2019 10:45:04.860 originate timestamp: e194afb7.7c01ea59 Fri, Dec 6 2019 10:45:11.484 transmit timestamp: e194afb7.7bf25b56 Fri, Dec 6 2019 10:45:11.484 delay 0.02589, dispersion 0.00000, offset 0.000051 6 Dec 10:45:11 ntpdate[3328]: no server suitable for synchronization found ---------- # ntpdate -4 -q -d 127.0.0.1 6 Dec 10:45:36 ntpdate[3329]: ntpdate 4.2.8p12-a (1) transmit(127.0.0.1) receive(127.0.0.1) 127.0.0.1: Server dropped: leap not in sync server 127.0.0.1, port 123 stratum 3, precision -23, leap 11, trust 000 refid [5.9.121.21], root delay 0.017548, root dispersion 2.071915 reference time: e194afb4.d40dab7a Fri, Dec 6 2019 10:45:08.828 originate timestamp: e194afd0.d54ad348 Fri, Dec 6 2019 10:45:36.833 transmit timestamp: e194afd0.d5431d47 Fri, Dec 6 2019 10:45:36.833 delay 0.02573, dispersion 0.00000, offset -0.000005 6 Dec 10:45:36 ntpdate[3329]: no server suitable for synchronization found ---------- https://share.riseup.net/#B_i8WV9SfSUSEJpRCun5rw >>> ktrace -d -i -f /var/my_ntpd/ktrace_output_ntpq.txt https://share.riseup.net/#Ox3sAwTrZUjdCWrbu11djQ >>> truss -o /var/my_ntpd/truss_output_ntpq.txt -a -f -d -D -e -H (In reply to Cy Schubert from comment #46) I've already updated the base version file and the man pages in base and the ports. There is no sense submitting patches. (In reply to Cy Schubert from comment #54) https://svnweb.freebsd.org/base/head/contrib/ntp/ntpd/ntp.conf.5man?view=markup#l3106 it still points out that 32mb is the default for non-Linux systems. so, upstream might be getting a patch for that or I just do not follow? (In reply to Vinícius Zavam from comment #55) See src r355388 and ports r519022. (In reply to Cy Schubert from comment #56) my bad though. sorry, I looked under /contrib/ntp instead of /usr/sbin/ntp. thanks! (In reply to Vinícius Zavam from comment #57) I meant to write 'base/usr.sbin/ntp' here (sorry for all the SPAM, just want to clarify) I've asked our upstream ntp.org to look at this PR. They have asked about the packets in your rawstats file that contain the "Unexpected origin timestamp 0xe180e455.d15bf3fb". They say that the message you see indicates that you are getting a 2nd incoming packet with the same origin timestamp, and this message is alerting you to this situation. Whether or not this PR is valid depends on what's going on with the root cause of your "Unexpected origin timestamp" message. (In reply to Cy Schubert from comment #59) I will start the test scenario again (using the memstick live image) and collect you these stats. by the end of the day I should mail them to you. thanks! (In reply to Cy Schubert from comment #59) incoming. hello there again, a quick update on this one: - someone pointed that iDRAC settings could interfere on networking stuffs on these kind of hardware - but I was not that convinced; anyway I did reset *ALL* BIOS and iDRAC settings to its defaults and got nothing new. still having issues :) Rawstats doesn't look right. There is consistency to the problem. Can you post the output of: sysctl kern.timecounter sysctl kern.eventtimer sysctl kern.clockrate sysctl hw.clockrate sysctl kern.hz After the system has been up for a day: date. List all the contents of rc.conf. Also, cat /var/db/ntp/ntpd.drift as I did restart one of the ntpd processes on a stable/12 machine yesterday, I can share its information now (after posting this, I should setup a lab with the same hardware running HEAD and get the data tomorrow).
FreeBSD 12.1-STABLE r354893 amd64 1201502
% cat /var/db/ntp/ntpd.drift
134.745
hw.clockrate: 1900
kern.clockrate: { hz = 1000, tick = 1000, profhz = 8128, stathz = 127 }
kern.eventtimer.periodic: 0
kern.eventtimer.timer: LAPIC
kern.eventtimer.idletick: 0
kern.eventtimer.singlemul: 2
kern.eventtimer.choice: LAPIC(600) HPET(550) HPET1(440) HPET2(440) i8254(100) RTC(0)
kern.eventtimer.et.HPET2.quality: 440
kern.eventtimer.et.HPET2.frequency: 24000000
kern.eventtimer.et.HPET2.flags: 3
kern.eventtimer.et.HPET1.quality: 440
kern.eventtimer.et.HPET1.frequency: 24000000
kern.eventtimer.et.HPET1.flags: 3
kern.eventtimer.et.HPET.quality: 550
kern.eventtimer.et.HPET.frequency: 24000000
kern.eventtimer.et.HPET.flags: 7
kern.eventtimer.et.i8254.quality: 100
kern.eventtimer.et.i8254.frequency: 1193182
kern.eventtimer.et.i8254.flags: 1
kern.eventtimer.et.RTC.quality: 0
kern.eventtimer.et.RTC.frequency: 32768
kern.eventtimer.et.RTC.flags: 17
kern.eventtimer.et.LAPIC.quality: 600
kern.eventtimer.et.LAPIC.frequency: 1900000000
kern.eventtimer.et.LAPIC.flags: 7
kern.hz: 1000
kern.timecounter.tsc_shift: 1
kern.timecounter.smp_tsc_adjust: 0
kern.timecounter.smp_tsc: 1
kern.timecounter.invariant_tsc: 1
kern.timecounter.fast_gettime: 1
kern.timecounter.tick: 1
kern.timecounter.choice: ACPI-fast(900) HPET(950) i8254(0) TSC(1000) dummy(-1000000)
kern.timecounter.hardware: TSC
kern.timecounter.alloweddeviation: 5
kern.timecounter.timehands_count: 2
kern.timecounter.stepwarnings: 0
kern.timecounter.tc.ACPI-fast.quality: 900
kern.timecounter.tc.ACPI-fast.frequency: 3579545
kern.timecounter.tc.ACPI-fast.counter: 1271069
kern.timecounter.tc.ACPI-fast.mask: 16777215
kern.timecounter.tc.HPET.quality: 950
kern.timecounter.tc.HPET.frequency: 24000000
kern.timecounter.tc.HPET.counter: 1219858571
kern.timecounter.tc.HPET.mask: 4294967295
kern.timecounter.tc.i8254.quality: 0
kern.timecounter.tc.i8254.frequency: 1193182
kern.timecounter.tc.i8254.counter: 49961
kern.timecounter.tc.i8254.mask: 65535
kern.timecounter.tc.TSC.quality: 1000
kern.timecounter.tc.TSC.frequency: 1900000000
kern.timecounter.tc.TSC.counter: 1044650998
kern.timecounter.tc.TSC.mask: 4294967295
(In reply to Vinícius Zavam from comment #65) running since Dec 16 18:11 CET should you guys need graphs of this particular machine's status (stable/12@r354893) please check https://imgur.com/a/RsHZZiw Capture some data: ntpq -p and list that here. Stop ntpd. Clear the ntpd.drift file. Run ntpdate. Start ntpd. Capture data after 30 minutes: ntpq -p Let it run for 24 hours. Capture data: ntpq -p (In reply to Cy Schubert from comment #68) # date ; ntpq -4 -n -c rl -wp localhost ; service ntpd stop ; rm /var/db/ntp/ntpd.drift ; ntpdate ntp2.prod.localdomain.de ; service ntpd start ; sleep 1800 ; date ; ntpq -4 -n -c rl -wp localhost ---------- Tue Dec 17 14:49:28 CET 2019 associd=0 status=c613 leap_alarm, sync_ntp, 1 event, spike_detect, version="ntpd 4.2.8p12-a (1)", processor="amd64", system="FreeBSD/12.1-STABLE", leap=11, stratum=3, precision=-23, rootdelay=6.309, rootdisp=1096.697, refid=10.0.5.3, reftime=e1a35a1d.493a2f96 Tue, Dec 17 2019 14:43:57.286, clock=e1a35b68.c0de8af2 Tue, Dec 17 2019 14:49:28.753, peer=36487, tc=6, mintc=3, offset=0.000000, frequency=134.745, sys_jitter=272.750727, clk_jitter=0.000, clk_wander=0.000, tai=37, leapsec=201701010000, expire=202006280000 remote refid st t when poll reach delay offset jitter ============================================================================== *10.0.5.3 213.136.0.252 2 u 9 64 17 0.266 533.253 412.658 +10.0.5.4 192.53.103.103 2 u 13 64 7 0.108 397.921 461.285 +10.0.32.248 213.136.0.252 2 u 5 64 17 1.026 1103.46 589.876 -10.0.32.249 213.136.0.252 2 u 13 64 7 1.022 806.084 394.811 Stopping ntpd. Waiting for PIDS: 72250, 72250. 17 Dec 14:49:37 ntpdate[24764]: step time server 10.0.5.4 offset 1.271688 sec Starting ntpd. ---------- in 30min I should return back to paste the result of the second 'ntpq -4 -n -c rl -wp localhost' Wow that's a large offset and jitter.
It should look more like this:
remote refid st t when poll reach delay offset jitter
==============================================================================
0.freebsd.pool. .POOL. 16 p - 64 0 0.000 0.000 0.001
+s206-75-147-25. 192.168.10.254 2 u 3 128 377 31.505 0.469 0.650
*ntp1.torix.ca .PTP0. 1 u 65 64 377 63.519 0.246 1.018
+nms.switch.ca 206.108.0.131 2 u 63 128 377 28.519 -0.945 0.667
-bhs2.vxnd.net 206.108.0.133 2 u 129 128 377 82.168 7.876 1.091
-draco.spiderspa 213.251.128.249 2 u 7 256 377 83.962 5.603 0.434
or this:
remote refid st t when poll reach delay offset jitter
==============================================================================
0.freebsd.pool. .POOL. 16 p - 64 0 0.000 0.000 0.000
*cwfw 206.108.0.131 2 u 44 64 377 0.295 -0.384 0.282
+cwsys 10.1.1.254 3 s 49 64 376 0.379 -0.699 0.197
+bob 10.1.1.254 3 s 10 64 377 0.441 -0.507 0.249
-nms.switch.ca 206.108.0.131 2 u 40 64 377 28.827 -1.526 0.339
-162.159.200.1 10.28.8.252 3 u 40 64 377 11.723 2.663 0.846
LAN or upstream network connectivity looks to be a problem.
ping -c 10 each of your upstream ntp servers (though I suspect they block ICMP ECHO) but worth testing anyway. You could try a traceroute instead.
Tue Dec 17 15:19:40 CET 2019
associd=0 status=c018 leap_alarm, sync_unspec, 1 event, no_sys_peer,
version="ntpd 4.2.8p12-a (1)", processor="amd64",
system="FreeBSD/12.1-STABLE", leap=11, stratum=16, precision=-22,
rootdelay=0.000, rootdisp=3.015, refid=STEP, reftime=(no time),
clock=e1a3627c.39550433 Tue, Dec 17 2019 15:19:40.223, peer=0, tc=6,
mintc=3, offset=0.000000, frequency=500.000, sys_jitter=0.000238,
clk_jitter=0.000, clk_wander=0.000, tai=37, leapsec=201701010000,
expire=202006280000
remote refid st t when poll reach delay offset jitter
==============================================================================
10.0.32.248 134.130.4.17 2 u 14 64 1 1.022 202.981 274.644
10.0.5.4 213.136.0.252 2 u 13 64 1 0.113 435.020 149.384
10.0.32.249 213.136.0.252 2 u 13 64 1 0.990 203.009 347.601
# cat /var/db/ntp/ntpd.drift
cat: /var/db/ntp/ntpd.drift: No such file or directory
(In reply to Cy Schubert from comment #70) yes, it is indeed! if I do check the same on my other machines it looks like this: % ntpq -4 -n -c rl -wp localhost associd=0 status=0618 leap_none, sync_ntp, 1 event, no_sys_peer, version="ntpd 4.2.8p12-a (1)", processor="amd64", system="FreeBSD/12.1-RELEASE-p1", leap=00, stratum=2, precision=-24, rootdelay=10.586, rootdisp=25.481, refid=192.53.103.108, reftime=e1a36486.c3041de6 Tue, Dec 17 2019 15:28:22.761, clock=e1a3671a.594f2eaf Tue, Dec 17 2019 15:39:22.348, peer=41868, tc=10, mintc=3, offset=-0.237943, frequency=24.158, sys_jitter=0.115137, clk_jitter=0.149, clk_wander=0.004, tai=37, leapsec=201701010000, expire=202006280000 remote refid st t when poll reach delay offset jitter ============================================================================== -134.130.4.17 .MRS. 1 u 278 1024 377 2.505 -0.233 0.203 +134.130.5.17 .MRS. 1 u 237 1024 377 2.310 -0.159 0.311 +129.143.2.23 .GPS. 1 u 203 1024 377 7.585 -0.202 0.257 -129.143.2.33 .PZF. 1 u 654 1024 377 7.484 -0.574 0.460 -192.87.106.2 .GPS. 1 u 895 1024 377 6.320 -0.071 0.762 *192.53.103.108 .PTB. 1 u 660 1024 377 10.586 -0.288 1.472 +192.53.103.103 .PTB. 1 u 214 1024 377 10.306 -0.263 1.027 +213.136.0.252 .PPS. 1 u 745 1024 377 6.285 -0.110 0.578 10.0.5.3 134.130.4.17 2 s 874 1024 376 0.181 -0.022 0.289 ---------- % ntpq -4 -n -c rl -wp localhost associd=0 status=0615 leap_none, sync_ntp, 1 event, clock_sync, version="ntpd 4.2.8p12-a (1)", processor="amd64", system="FreeBSD/11.2-STABLE", leap=00, stratum=2, precision=-24, rootdelay=10.842, rootdisp=27.292, refid=192.53.103.103, reftime=e1a36434.8416e548 Tue, Dec 17 2019 15:27:00.515, clock=e1a36735.530754d7 Tue, Dec 17 2019 15:39:49.324, peer=31263, tc=10, mintc=3, offset=0.054371, frequency=73.858, sys_jitter=0.039632, clk_jitter=0.075, clk_wander=0.005, tai=37, leapsec=201701010000, expire=202006280000 remote refid st t when poll reach delay offset jitter ============================================================================== -134.130.4.17 .MRS. 1 u 747 1024 377 2.697 -0.008 4.593 +134.130.5.17 .MRS. 1 u 159 1024 377 2.704 0.005 3.223 -129.143.2.23 .GPS. 1 u 642 1024 377 7.911 -0.033 0.441 -129.143.2.33 .PZF. 1 u 312 1024 377 7.862 -0.325 0.061 +192.87.106.2 .GPS. 1 u 949 1024 377 6.713 0.119 0.065 -192.53.103.108 .PTB. 1 u 872 1024 377 14.247 1.696 7.276 *192.53.103.103 .PTB. 1 u 769 1024 377 10.842 0.035 0.303 -213.136.0.252 .PPS. 1 u 733 1024 377 6.967 0.038 0.078 10.0.32.249 213.136.0.252 2 s 935 1024 376 0.120 -0.066 0.103 (In reply to Cy Schubert from comment #70) 2 of the servers you see are in the same VLAN; there's also no firewall filtering this kind of thing (not even scrubbing udp/123) for the other 2. you can check the 'delay' column, but anyway ... % ping -c 10 10.0.5.3 | grep -B1 avg 10 packets transmitted, 10 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 0.183/0.402/1.267/0.309 ms % ping -c 10 10.0.32.248 | grep -B1 avg 10 packets transmitted, 10 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 0.958/1.024/1.157/0.064 ms (In reply to Vinícius Zavam from comment #71) Don't delete the file, just clear it (as in empty it). (In reply to Vinícius Zavam from comment #72) Can we see the ntpd.drift file for the two servers that do work, please? (In reply to Vinícius Zavam from comment #71) Did you run ntpdate prior to starting ntpd? Let me ask again, please. What is in rc.conf? Can you list it, please? (In reply to Cy Schubert from comment #76) I'm curious about this one, Cy. This used to be recommended practice before we had the -g option to ntpd, but we're 95%+ comfortable saying that it's no longer needed. I've had one person say they know cases where it matters, but I haven't seen the supporting information yet. (In reply to Cy Schubert from comment #70) That's not a large offset/jitter given that only 3 or 4 polls have occurred. Look at the 'reach' indicator. 7 means only 3 polls have happened, and 17 means only 4 polls have happened. I'm betting the 'iburst' flag is not being provided. We recommend using 'iburst' because it will sync your clock sooner. (In reply to Harlan Stenn from comment #79) If you *are* using iburst, then you're running 'ntpq' about 5 seconds after starting ntpd. Wait at least 15 seconds. iburst sends up to 8 packets at 2 second intervals. It takes at least 5 samples for a warm-start to settle, which usually happens in about 11 seconds' time. (In reply to Harlan Stenn from comment #78) You are correct Harlan. -g will work. (In reply to Harlan Stenn from comment #80) Yes, he's specified iburst on: pool 0.freebsd.pool.ntp.org iburst (In reply to Cy Schubert from comment #75) egypcio@ntp1:~ % uptime 10:33AM up 5 days, 18:13, 4 users, load averages: 0.15, 0.14, 0.15 egypcio@ntp1:~ % cat /var/db/ntpd.drift 23.727 egypcio@ntp2:~ % uptime 10:33AM up 6 days, 18:30, 3 users, load averages: 0.12, 0.11, 0.09 egypcio@ntp2:~ % cat /var/db/ntpd.drift 24.216 (In reply to Cy Schubert from comment #76) yes, I did. as we see on Comment 69 this was the command line I ran: # date ; ntpq -4 -n -c rl -wp localhost ; service ntpd stop ; rm /var/db/ntp/ntpd.drift ; ntpdate ntp2.prod.localdomain.de ; service ntpd start ; sleep 1800 ; date ; ntpq -4 -n -c rl -wp localhost (In reply to Cy Schubert from comment #74) the ntpd creates a new one anyway (that's what I did see on its manpage). should not bother. right? btw, here is the current status of the drift file on the stable/12: egypcio@stable12:~ % ls -lash /var/db/ntp/ntpd.pid 4 -rw-r--r-- 1 ntpd ntpd 5B Dec 17 14:49 /var/db/ntp/ntpd.pid egypcio@stable12:~ % cat /var/db/ntp/ntpd.drift 500.000 (In reply to Cy Schubert from comment #77) here is the rc.conf on the stable12 machine: keymap="de" keyrate="fast" saver="blank" hostname="stable12.prod.localdomain.de" cloned_interfaces="gif21 gif22 gif31 gif32" vlans_bge0="vlan21 vlan22 vlan23 vlan24 vlan249" vlans_bge1="vlan3011" create_args_vlan21="vlan 21" create_args_vlan22="vlan 22" create_args_vlan23="vlan 23" create_args_vlan24="vlan 24" create_args_vlan249="vlan 249" create_args_vlan3011="vlan 3011" ifconfig_bge0="inet 10.0.5.7/24 -lro -tso -rxcsum -txcsum" ifconfig_bge1="up -lro -tso -rxcsum -txcsum" ifconfig_enc0="up" ifconfig_vlan21="inet 10.0.21.251/24" ifconfig_vlan22="inet 10.0.22.251/24" ifconfig_vlan23="inet 10.0.23.251/24" ifconfig_vlan24="inet 10.0.24.251/24" ifconfig_vlan249="inet 10.0.249.5/29" ifconfig_vlan3011="inet 192.168.123.12/24" . /etc/rc.conf.carp.vlan3011 ifconfig_bge0_alias0="inet 10.0.5.254/32 vhid 210 pass jurubeba advskew 232" ifconfig_vlan21_alias0="inet 10.0.21.254/32 vhid 21 pass jurubeba advskew 232" ifconfig_vlan22_alias0="inet 10.0.22.254/32 vhid 22 pass jurubeba advskew 232" ifconfig_vlan23_alias0="inet 10.0.23.253/32 vhid 23 pass jurubeba advskew 232" ifconfig_vlan24_alias0="inet 10.0.24.254/32 vhid 24 pass jurubeba advskew 232" ifconfig_vlan249_alias0="inet 10.0.249.1/29 vhid 250 pass jurubeba advskew 232" ifconfig_gif21="10.0.248.26/30 10.0.248.25 tunnel 192.168.123.12 192.168.99.59 mtu 1416 description 'IPSec VPN to abc2a'" ifconfig_gif22="10.0.248.10/30 10.0.248.9 tunnel 192.168.123.12 172.16.26.52 mtu 1416 description 'IPSec VPN to abc2b'" ifconfig_gif32="10.0.248.46/30 10.0.248.45 tunnel 192.168.123.12 172.31.41.8 mtu 1416 description 'IPSec VPN to qwerty2a'" gateway_enable="YES" defaultrouter="192.168.123.1" ipv6_activate_all_interfaces="YES" ipv6_gateway_enable="YES" ifconfig_bge0_ipv6="inet6 fd00:0:0:5::7/64" ifconfig_bge0_alias1="inet6 fd00:0:0:5::254/64 vhid 210" ifconfig_vlan249_ipv6="inet6 fd00:0:0:249::5/64" ifconfig_vlan249_alias1="inet6 fd00:0:0:249::1/64 vhid 250" ifconfig_vlan3011_ipv6="inet6 2001:db8:fb5d::12/64" ipv6_defaultrouter="fe80::1%vlan3011" powerd_enable="YES" powerd_flags="-a maximum -b hiadaptive" ntpdate_enable="YES" ntpd_enable="YES" ntpd_flags="${ntpd_flags} -4 --logfile=/var/log/ntpd.log --statsdir=/etc/ntp/stats" ntpd_sync_on_start="YES" sshd_enable="YES" syslogd_flags="-s -4" postfix_enable="YES" sendmail_enable="NONE" sendmail_msp_queue_enable="NO" sendmail_outbound_enable="NO" sendmail_submit_enable="NO" quagga_enable="YES" quagga_daemons="zebra ospfd" pf_enable="YES" pflog_enable="YES" pfsync_enable="YES" pfsync_syncdev="vlan249" pfsync_syncpeer="10.0.249.3" ipsec_enable="YES" racoon_enable="YES" racoon_flags="-l /var/log/racoon.log" ossec_hids_enable="YES" zabbix_agentd_enable="YES" nrpe3_enable="YES" I left the live image running for about 5 days, and here are the results:
root@head-r355121-amd64:~ # uptime
9:50AM up 5 days, 19:56, 2 users, load averages: 0.09, 0.22, 0.23
root@head-r355121-amd64:~ # ps afxuww | grep ntp
root 3067 0.0 0.0 19308 6352 - Ss Tue13 0:22.59 /usr/sbin/ntpd --logfile=/var/log/ntpd.log --statsdir=/var/db/ntp/stats -p /var/db/ntp/ntpd.pid -c /var/etc_ntp.conf -f /var/db/ntp/ntpd.drift -4 -g
root 16950 0.0 0.0 11364 2620 0 S+ 09:50 0:00.00 grep ntp
root@head-r355121-amd64:~ # cat /var/db/ntp/ntpd.drift
500.000
root@head-r355121-amd64:~ # ntpq -4 -n -c rl -wp localhost
associd=0 status=c613 leap_alarm, sync_ntp, 1 event, spike_detect,
version="ntpd 4.2.8p12-a (1)", processor="amd64",
system="FreeBSD/13.0-CURRENT", leap=11, stratum=3, precision=-23,
rootdelay=23.848, rootdisp=1548.403, refid=80.237.128.148,
reftime=e1ab0c2b.f6e024b2 Mon, Dec 23 2019 9:49:31.964,
clock=e1ab0c60.d73bbaf7 Mon, Dec 23 2019 9:50:24.840, peer=28700, tc=6,
mintc=3, offset=0.000000, frequency=500.000, sys_jitter=87.612594,
clk_jitter=0.000, clk_wander=0.000, leapsec=201701010000,
expire=201912280000
remote refid st t when poll reach delay offset jitter
==============================================================================
0.freebsd.pool.ntp.org
.POOL. 16 p - 64 0 0.000 0.000 0.000
83.68.137.76 131.130.251.107 2 u 53 64 1 16.140 323.987 192.995
5.39.184.5 46.243.26.34 2 u 48 64 1 7.025 321.914 201.789
212.18.3.19 212.18.1.106 2 u 16 64 1 10.195 322.169 252.769
82.64.45.50 .GPS. 1 u 15 64 1 32.063 541.354 140.598
88.99.216.194 131.188.3.223 2 u 15 64 1 8.995 659.318 214.715
82.100.248.10 131.188.3.222 2 u 50 64 1 4.476 481.363 114.111
162.159.200.123 10.48.8.4 3 u 48 64 1 1.917 594.497 184.049
145.239.0.227 131.188.3.223 2 u 35 64 1 9.819 621.196 194.087
162.159.200.1 10.48.8.4 3 u 46 64 1 1.897 485.164 116.870
116.203.149.9 131.188.3.222 2 u 35 64 1 7.630 504.075 123.998
*80.237.128.148 162.23.41.10 2 u 53 64 1 7.964 587.071 182.109
144.76.60.190 131.188.3.221 2 u 24 64 1 9.123 638.931 208.746
root@head-r355121-amd64:~ # cat /etc/rc.conf
sendmail_enable="NONE"
hostid_enable="NO"
root@head-r355121-amd64:~ # grep -v ^# /var/etc_ntp.conf
leapfile "/var/db/ntpd.leap-seconds.list"
pool 0.freebsd.pool.ntp.org iburst
restrict 127.0.0.1
restrict ::1
restrict default limited kod nomodify notrap noquery nopeer
restrict source limited kod nomodify notrap noquery
statistics clockstats cryptostats loopstats peerstats rawstats sysstats
tos minclock 3 maxclock 6
ntpd.log -> https://share.riseup.net/#QaFZ3JaDeJcEL3AX6gla9Q
(In reply to Vinícius Zavam from comment #83) quering the status and stuff of the 2 working servers (running 12.1R-p1): % ntpq -4 -n -c rl -wp ntp1 associd=0 status=0615 leap_none, sync_ntp, 1 event, clock_sync, version="ntpd 4.2.8p12-a (1)", processor="amd64", system="FreeBSD/12.1-RELEASE-p1", leap=00, stratum=2, precision=-24, rootdelay=5.898, rootdisp=17.291, refid=213.136.0.252, reftime=e1ab0bdd.8195ba5d Mon, Dec 23 2019 9:48:13.506, clock=e1ab0e6a.e53c3b38 Mon, Dec 23 2019 9:59:06.895, peer=37729, tc=8, mintc=3, offset=0.362365, frequency=23.796, sys_jitter=0.051579, clk_jitter=0.140, clk_wander=0.020, tai=37, leapsec=201701010000, expire=202006280000 remote refid st t when poll reach delay offset jitter ============================================================================== -134.130.4.17 .MRS. 1 u 164 256 377 2.231 0.337 0.187 -134.130.5.17 .MRS. 1 u 232 256 377 2.212 0.501 1.212 -129.143.2.23 .GPS. 1 u 112 256 377 7.405 0.283 0.215 -129.143.2.33 .PZF. 1 u 131 256 377 7.482 -0.020 1.797 -192.87.106.2 .GPS. 1 u 242 256 377 5.972 0.664 0.128 +192.53.103.108 .PTB. 1 u 188 256 377 10.362 0.302 1.297 +192.53.103.103 .PTB. 1 u 127 256 377 10.424 0.335 1.398 *213.136.0.252 .PPS. 1 u 121 256 377 5.898 0.382 0.113 10.0.5.4 134.130.4.17 2 s 174 256 376 0.138 0.582 0.082 ---------- % ntpq -4 -n -c rl -wp ntp2 associd=0 status=0618 leap_none, sync_ntp, 1 event, no_sys_peer, version="ntpd 4.2.8p12-a (1)", processor="amd64", system="FreeBSD/12.1-RELEASE-p1", leap=00, stratum=2, precision=-24, rootdelay=2.515, rootdisp=19.147, refid=134.130.4.17, reftime=e1ab0c8c.c0f35810 Mon, Dec 23 2019 9:51:08.753, clock=e1ab0e71.f26232e3 Mon, Dec 23 2019 9:59:13.946, peer=41863, tc=10, mintc=3, offset=-0.054842, frequency=24.215, sys_jitter=0.118919, clk_jitter=0.077, clk_wander=0.011, tai=37, leapsec=201701010000, expire=202006280000 remote refid st t when poll reach delay offset jitter ============================================================================== *134.130.4.17 .MRS. 1 u 485 512 377 2.515 -0.057 0.121 -134.130.5.17 .MRS. 1 u 139 1024 377 2.369 -0.208 0.987 -129.143.2.23 .GPS. 1 u 983 1024 277 7.655 -0.276 0.073 -129.143.2.33 .PZF. 1 u 79 1024 377 7.541 -0.425 0.083 +192.87.106.2 .GPS. 1 u 235 1024 377 5.548 -0.080 0.110 -192.53.103.108 .PTB. 1 u 270 1024 377 10.481 -0.180 0.885 -192.53.103.103 .PTB. 1 u 298 1024 377 10.411 -0.177 1.438 +213.136.0.252 .PPS. 1 u 355 1024 377 6.282 -0.030 0.195 10.0.5.3 213.136.0.252 2 s 172 256 377 0.144 -0.585 0.084 (In reply to Cy Schubert from comment #81) which is being used already (In reply to Vinícius Zavam from comment #85) a bit more from the stable/12 machine: % ntpq -4 -n -c rl localhost associd=0 status=c018 leap_alarm, sync_unspec, 1 event, no_sys_peer, version="ntpd 4.2.8p12-a (1)", processor="amd64", system="FreeBSD/12.1-STABLE", leap=11, stratum=16, precision=-22, rootdelay=0.000, rootdisp=1.020, refid=STEP, reftime=(no time), clock=e1ab0f74.8d96c57d Mon, Dec 23 2019 10:03:32.553, peer=0, tc=6, mintc=3, offset=0.000000, frequency=500.000, sys_jitter=0.000238, clk_jitter=0.000, clk_wander=0.000, tai=37, leapsec=201701010000, expire=202006280000 (In reply to Vinícius Zavam from comment #90) https://imgur.com/a/6p1tTue What is the output of ntptime? (In reply to Cy Schubert from comment #92) root@freebsd-HEAD-r356310-amd64:~ # ntptime ntp_gettime() returns code 5 (ERROR) time e1bd7b28.09d56000 Mon, Jan 6 2020 9:23:52.038, (.690038412), maximum error 16453500 us, estimated error 16000000 us, TAI offset 0 ntp_adjtime() returns code 5 (ERROR) modes 0x0 (), offset 0.000 us, frequency 500.000 ppm, interval 1 s, maximum error 16453500 us, estimated error 16000000 us, status 0x41 (PLL,UNSYNC), time constant 3, precision 0.000 us, tolerance 496 ppm, root@freebsd-HEAD-r356310-amd64:~ # ps afxuww | grep ntp root 3786 0.0 0.0 19108 6416 - Ss 09:08 0:00.07 /usr/sbin/ntpd --logfile=/var/log/ntpd.log --statsdir=/var/db/ntp/stats -p /var/db/ntp/ntpd.pid -c /var/etc_ntp.conf -f /var/db/ntp/ntpd.drift -4 -g root 3824 0.0 0.0 11376 2632 0 S+ 09:23 0:00.00 grep ntp root@freebsd-HEAD-r356310-amd64:~ # ntpq -4 -n -c rl -wp localhost associd=0 status=0613 leap_none, sync_ntp, 1 event, spike_detect, version="ntpd 4.2.8p12-a (1)", processor="amd64", system="FreeBSD/13.0-CURRENT", leap=00, stratum=3, precision=-23, rootdelay=26.166, rootdisp=537.015, refid=5.199.135.170, reftime=e1bd7ae3.9a3374c2 Mon, Jan 6 2020 9:22:43.602, clock=e1bd7b01.85243efb Mon, Jan 6 2020 9:23:13.520, peer=46567, tc=6, mintc=3, offset=73.042339, frequency=500.000, sys_jitter=52.538034, clk_jitter=25.824, clk_wander=0.000, tai=37, leapsec=201701010000, expire=202006280000 remote refid st t when poll reach delay offset jitter ============================================================================== 0.freebsd.pool.ntp.org .POOL. 16 p - 64 0 0.000 0.000 0.000 85.25.148.4 141.2.22.74 4 u 7 64 1 7.366 331.053 183.178 88.99.216.194 131.188.3.223 2 u 9 64 1 9.158 326.029 176.823 213.239.239.166 205.46.178.169 2 u 8 64 1 18.858 321.180 176.054 212.18.3.18 212.18.1.106 2 u 5 64 1 10.313 331.719 178.953 +159.69.150.81 131.188.3.220 2 u 44 64 1 7.864 259.263 145.789 +88.198.17.248 212.18.3.18 3 u 30 64 1 24.399 277.903 190.095 +85.220.190.246 213.172.96.14 2 u 28 64 1 4.689 289.230 201.493 +78.46.90.21 129.69.1.153 2 u 29 64 1 9.562 287.327 201.722 *5.199.135.170 131.188.3.222 2 u 30 64 1 17.117 277.570 188.922 +116.203.151.74 131.188.3.222 2 u 29 64 1 7.700 287.364 195.871 +176.9.1.211 122.227.206.195 3 u 28 64 1 9.144 288.973 201.673 +62.116.130.3 192.53.103.104 2 u 85 64 1 9.756 185.961 114.962 root@freebsd-HEAD-r356310-amd64:~ # uname -rmpsiUK FreeBSD 13.0-CURRENT amd64 amd64 GENERIC 1300073 1300073 root@freebsd-HEAD-r356310-amd64:~ # ll /var/db/ntp/ntpd.pid -rw-r--r-- 1 ntpd ntpd 4 Jan 6 09:08 /var/db/ntp/ntpd.pid root@freebsd-HEAD-r356310-amd64:~ # date Mon Jan 6 09:23:26 UTC 2020 root@freebsd-HEAD-r356310-amd64:~ # uptime 9:23AM up 28 mins, 2 users, load averages: 0.17, 0.19, 0.16 (In reply to Vinícius Zavam from comment #93) here, 'ntptime' output from my ntp1 and ntp2 boxes (12.1-Rp1): egypcio@ntp1:~ % ntptime ntp_gettime() returns code 0 (OK) time e1bdb268.5d6c508c Mon, Jan 6 2020 14:19:36.364, (.364934606), maximum error 240727 us, estimated error 192 us, TAI offset 37 ntp_adjtime() returns code 0 (OK) modes 0x0 (), offset -1501.932 us, frequency 23.589 ppm, interval 1 s, maximum error 240727 us, estimated error 192 us, status 0x2001 (PLL,NANO), time constant 9, precision 1.000 us, tolerance 496 ppm, egypcio@ntp2:~ % ntptime ntp_gettime() returns code 0 (OK) time e1bdb26c.9baf6408 Mon, Jan 6 2020 14:19:40.608, (.608145871), maximum error 114455 us, estimated error 84 us, TAI offset 37 ntp_adjtime() returns code 0 (OK) modes 0x0 (), offset -382.554 us, frequency 24.064 ppm, interval 1 s, maximum error 114455 us, estimated error 84 us, status 0x2001 (PLL,NANO), time constant 8, precision 1.000 us, tolerance 496 ppm, silly question: as this issue definitely seems to also be influenced by the hardware we are using, and not only the FreeBSD version/revision, what other information would you need to help debugging it? - I mentioned the CPU and also server model before. Yes, ntptime tells us this. On only the affected system, sysctl -a | egrep 'timecounter|eventtimer' (In reply to Cy Schubert from comment #96) root@head-r356310-amd64:~ # sysctl -a | egrep 'timecounter|eventtimer' kern.timecounter.tsc_shift: 1 kern.timecounter.smp_tsc_adjust: 0 kern.timecounter.smp_tsc: 1 kern.timecounter.invariant_tsc: 1 kern.timecounter.fast_gettime: 1 kern.timecounter.tick: 1 kern.timecounter.choice: ACPI-fast(900) HPET(950) i8254(0) TSC(1000) dummy(-1000000) kern.timecounter.hardware: TSC kern.timecounter.alloweddeviation: 5 kern.timecounter.timehands_count: 2 kern.timecounter.stepwarnings: 0 kern.timecounter.tc.ACPI-fast.quality: 900 kern.timecounter.tc.ACPI-fast.frequency: 3579545 kern.timecounter.tc.ACPI-fast.counter: 16329298 kern.timecounter.tc.ACPI-fast.mask: 16777215 kern.timecounter.tc.HPET.quality: 950 kern.timecounter.tc.HPET.frequency: 24000000 kern.timecounter.tc.HPET.counter: 1316611790 kern.timecounter.tc.HPET.mask: 4294967295 kern.timecounter.tc.i8254.quality: 0 kern.timecounter.tc.i8254.frequency: 1193182 kern.timecounter.tc.i8254.counter: 45741 kern.timecounter.tc.i8254.mask: 65535 kern.timecounter.tc.TSC.quality: 1000 kern.timecounter.tc.TSC.frequency: 1900000000 kern.timecounter.tc.TSC.counter: 307319750 kern.timecounter.tc.TSC.mask: 4294967295 kern.eventtimer.periodic: 0 kern.eventtimer.timer: LAPIC kern.eventtimer.idletick: 0 kern.eventtimer.singlemul: 2 kern.eventtimer.choice: LAPIC(600) HPET(550) HPET1(440) HPET2(440) i8254(100) RTC(0) kern.eventtimer.et.HPET2.quality: 440 kern.eventtimer.et.HPET2.frequency: 24000000 kern.eventtimer.et.HPET2.flags: 3 kern.eventtimer.et.HPET1.quality: 440 kern.eventtimer.et.HPET1.frequency: 24000000 kern.eventtimer.et.HPET1.flags: 3 kern.eventtimer.et.HPET.quality: 550 kern.eventtimer.et.HPET.frequency: 24000000 kern.eventtimer.et.HPET.flags: 7 kern.eventtimer.et.i8254.quality: 100 kern.eventtimer.et.i8254.frequency: 1193182 kern.eventtimer.et.i8254.flags: 1 kern.eventtimer.et.RTC.quality: 0 kern.eventtimer.et.RTC.frequency: 32768 kern.eventtimer.et.RTC.flags: 17 kern.eventtimer.et.LAPIC.quality: 600 kern.eventtimer.et.LAPIC.frequency: 1900000000 kern.eventtimer.et.LAPIC.flags: 7 Created attachment 210496 [details]
ntpdate_offsetvars.txt
I left the machine running (for about 16h) with a little script running in background to check the offset variation of its clock "ticking" WITHOUT RUNNING `ntpd`.
here is the code:
#!/bin/sh
while true; do
ntpdate -qp1 ntp1.netcologne.de
sleep 60
done
attached you have the output.
What is the output of, sysctl dev.cpu.0.cx_lowest Actually, sysctl -a | grep cx_lowest (In reply to Cy Schubert from comment #100) % sysctl -a | grep cx_lowest hw.acpi.cpu.cx_lowest: C1 dev.cpu.5.cx_lowest: C1 dev.cpu.4.cx_lowest: C1 dev.cpu.3.cx_lowest: C1 dev.cpu.2.cx_lowest: C1 dev.cpu.1.cx_lowest: C1 dev.cpu.0.cx_lowest: C1 Can you rebuild your kernel without EARLY_AP_STARTUP ? (In reply to Cy Schubert from comment #102) sure! I should start building in a few, again I will use the 'release.sh' and get a bootable USB image to try it out - will take a while. (In reply to Vinícius Zavam from comment #103) # /usr/src/sys/amd64/conf/PR242137 (HEAD@r356428) include GENERIC ident PR242137 nooptions EARLY_AP_STARTUP (In reply to Vinícius Zavam from comment #104) root@head-r356428-amd64:~ # uptime 11:24AM up 49 mins, 2 users, load averages: 0.15, 0.20, 0.21 root@head-r356428-amd64:~ # ps afxuww | grep ntp root 3287 0.0 0.0 19136 6420 - Ss 10:55 0:00.10 /usr/sbin/ntpd --logfile=/var/log/ntpd.log --statsdir=/var/db/ntp/stats -p /var/db/ntp/ntpd.pid -c /var/etc_ntp.conf -f /var/db/ntp/ntpd.drift -4 -g root 3321 0.0 0.0 11376 2600 0 S+ 11:24 0:00.00 grep ntp root@head-r356428-amd64:~ # ntpq -4 -n -c rl -wp localhost associd=0 status=c613 leap_alarm, sync_ntp, 1 event, spike_detect, version="ntpd 4.2.8p12-a (1)", processor="amd64", system="FreeBSD/13.0-CURRENT", leap=11, stratum=3, precision=-23, rootdelay=14.370, rootdisp=1525.451, refid=88.99.76.254, reftime=e1c03a04.fd3def83 Wed, Jan 8 2020 11:22:44.989, clock=e1c03a8d.1afc38e7 Wed, Jan 8 2020 11:25:01.105, peer=40336, tc=6, mintc=3, offset=0.000000, frequency=500.000, sys_jitter=93.026842, clk_jitter=0.000, clk_wander=0.000, tai=37, leapsec=201701010000, expire=202006280000 remote refid st t when poll reach delay offset jitter ============================================================================== 0.freebsd.pool.ntp.org .POOL. 16 p - 64 0 0.000 0.000 0.000 131.188.3.222 .MBGh. 1 u 45 64 1 7.803 442.385 261.983 144.76.76.107 205.46.178.169 2 u 46 64 1 8.998 444.277 258.226 162.159.200.123 10.48.8.4 3 u 43 64 1 1.893 443.045 265.364 159.69.150.81 131.188.3.220 2 u 41 64 1 7.715 838.467 260.863 80.127.152.30 131.211.8.244 2 u 30 64 1 9.257 855.442 268.080 91.209.0.17 232.6.188.111 2 u 16 64 1 17.080 657.616 134.005 91.220.110.117 236.118.239.31 2 u 13 64 1 56.410 773.421 192.055 195.219.205.9 195.219.14.21 2 u 14 64 1 15.401 440.598 315.054 +178.63.247.119 17.253.54.123 2 u 13 64 1 9.463 773.580 217.357 *88.99.76.254 17.253.54.125 2 u 12 64 1 9.289 545.807 193.492 193.141.27.6 192.53.103.108 2 u 10 64 1 6.720 551.751 196.067 141.30.228.4 212.18.3.18 3 u 9 64 1 20.020 445.637 276.813 root@head-r356428-amd64:~ # ntptime ntp_gettime() returns code 0 (OK) time e1c03a8e.e54c9860 Wed, Jan 8 2020 11:25:02.895, (.895700329), maximum error 925331 us, estimated error 0 us, TAI offset 37 ntp_adjtime() returns code 0 (OK) modes 0x0 (), offset 0.000 us, frequency 500.000 ppm, interval 1 s, maximum error 925331 us, estimated error 0 us, status 0x2001 (PLL,NANO), time constant 6, precision 1.000 us, tolerance 496 ppm, root@head-r356428-amd64:~ # sysctl -a | grep cx_lowest hw.acpi.cpu.cx_lowest: C1 dev.cpu.5.cx_lowest: C1 dev.cpu.4.cx_lowest: C1 dev.cpu.3.cx_lowest: C1 dev.cpu.2.cx_lowest: C1 dev.cpu.1.cx_lowest: C1 dev.cpu.0.cx_lowest: C1 root@head-r356428-amd64:~ # sysctl -a | egrep 'timecounter|eventtimer' kern.timecounter.tsc_shift: 1 kern.timecounter.smp_tsc_adjust: 0 kern.timecounter.smp_tsc: 1 kern.timecounter.invariant_tsc: 1 kern.timecounter.fast_gettime: 1 kern.timecounter.tick: 1 kern.timecounter.choice: TSC(1000) ACPI-fast(900) HPET(950) i8254(0) dummy(-1000000) kern.timecounter.hardware: TSC kern.timecounter.alloweddeviation: 5 kern.timecounter.timehands_count: 2 kern.timecounter.stepwarnings: 0 kern.timecounter.tc.TSC.quality: 1000 kern.timecounter.tc.TSC.frequency: 1900000000 kern.timecounter.tc.TSC.counter: 4137183290 kern.timecounter.tc.TSC.mask: 4294967295 kern.timecounter.tc.ACPI-fast.quality: 900 kern.timecounter.tc.ACPI-fast.frequency: 3579545 kern.timecounter.tc.ACPI-fast.counter: 11808359 kern.timecounter.tc.ACPI-fast.mask: 16777215 kern.timecounter.tc.HPET.quality: 950 kern.timecounter.tc.HPET.frequency: 24000000 kern.timecounter.tc.HPET.counter: 283868598 kern.timecounter.tc.HPET.mask: 4294967295 kern.timecounter.tc.i8254.quality: 0 kern.timecounter.tc.i8254.frequency: 1193182 kern.timecounter.tc.i8254.counter: 38880 kern.timecounter.tc.i8254.mask: 65535 kern.eventtimer.periodic: 0 kern.eventtimer.timer: LAPIC kern.eventtimer.idletick: 0 kern.eventtimer.singlemul: 2 kern.eventtimer.choice: LAPIC(600) HPET(550) HPET1(440) HPET2(440) i8254(100) RTC(0) kern.eventtimer.et.HPET2.quality: 440 kern.eventtimer.et.HPET2.frequency: 24000000 kern.eventtimer.et.HPET2.flags: 3 kern.eventtimer.et.HPET1.quality: 440 kern.eventtimer.et.HPET1.frequency: 24000000 kern.eventtimer.et.HPET1.flags: 3 kern.eventtimer.et.HPET.quality: 550 kern.eventtimer.et.HPET.frequency: 24000000 kern.eventtimer.et.HPET.flags: 7 kern.eventtimer.et.i8254.quality: 100 kern.eventtimer.et.i8254.frequency: 1193182 kern.eventtimer.et.i8254.flags: 1 kern.eventtimer.et.RTC.quality: 0 kern.eventtimer.et.RTC.frequency: 32768 kern.eventtimer.et.RTC.flags: 17 kern.eventtimer.et.LAPIC.quality: 600 kern.eventtimer.et.LAPIC.frequency: 1900000000 kern.eventtimer.et.LAPIC.flags: 7 (In reply to Vinícius Zavam from comment #98) I ran this test again, using ***12.1-RELEASE*** and here is the output of few queries: ----- server 81.173.193.222, stratum 2, offset 0.723093, delay 0.02759 8 Jan 12:43:56 ntpdate[3096]: step time server 81.173.193.222 offset 0.723093 sec server 81.173.193.222, stratum 2, offset 0.726101, delay 0.02786 8 Jan 12:44:56 ntpdate[3103]: step time server 81.173.193.222 offset 0.726101 sec server 81.173.193.222, stratum 2, offset 0.729273, delay 0.02727 8 Jan 12:45:56 ntpdate[3105]: step time server 81.173.193.222 offset 0.729273 sec server 81.173.193.222, stratum 2, offset 0.732274, delay 0.02727 8 Jan 12:46:56 ntpdate[3107]: step time server 81.173.193.222 offset 0.732274 sec server 81.173.193.222, stratum 2, offset 0.734884, delay 0.02863 8 Jan 12:47:56 ntpdate[3110]: step time server 81.173.193.222 offset 0.734884 sec server 81.173.193.222, stratum 2, offset 0.738171, delay 0.02786 8 Jan 12:48:56 ntpdate[3113]: step time server 81.173.193.222 offset 0.738171 sec server 81.173.193.222, stratum 2, offset 0.741373, delay 0.02745 8 Jan 12:49:57 ntpdate[3115]: step time server 81.173.193.222 offset 0.741373 sec server 81.173.193.222, stratum 2, offset 0.742197, delay 0.03177 8 Jan 12:50:57 ntpdate[3117]: step time server 81.173.193.222 offset 0.742197 sec server 81.173.193.222, stratum 2, offset 0.747455, delay 0.02739 8 Jan 12:51:57 ntpdate[3119]: step time server 81.173.193.222 offset 0.747455 sec server 81.173.193.222, stratum 2, offset 0.750519, delay 0.02797 8 Jan 12:52:57 ntpdate[3121]: step time server 81.173.193.222 offset 0.750519 sec server 81.173.193.222, stratum 2, offset 0.753328, delay 0.02788 8 Jan 12:53:57 ntpdate[3123]: step time server 81.173.193.222 offset 0.753328 sec server 81.173.193.222, stratum 2, offset 0.756456, delay 0.02759 8 Jan 12:54:58 ntpdate[3125]: step time server 81.173.193.222 offset 0.756456 sec server 81.173.193.222, stratum 2, offset 0.759400, delay 0.02742 8 Jan 12:55:58 ntpdate[3132]: step time server 81.173.193.222 offset 0.759400 sec server 81.173.193.222, stratum 2, offset 0.762648, delay 0.02757 8 Jan 12:56:58 ntpdate[3134]: step time server 81.173.193.222 offset 0.762648 sec server 81.173.193.222, stratum 2, offset 0.765475, delay 0.02742 8 Jan 12:57:58 ntpdate[3136]: step time server 81.173.193.222 offset 0.765475 sec server 81.173.193.222, stratum 2, offset 0.768636, delay 0.02747 8 Jan 12:58:58 ntpdate[3138]: step time server 81.173.193.222 offset 0.768636 sec server 81.173.193.222, stratum 2, offset 0.771571, delay 0.02805 8 Jan 12:59:58 ntpdate[3140]: step time server 81.173.193.222 offset 0.771571 sec ----- afaik, we can definable see the difference on how the "clock ticking" behaves. right? I mean: using HEAD the offset increases around 0.140207 per minute; with RELEASE/12.1.0 it increases 0.003008 % bc 16.447002 - 16.306795 .140207 0.726101 - 0.723093 .003008 Agreed, this is a timecounter issue, not an NTP issue. NTP will only adjust the clock up to 500 PPM, after which it stops. 500 PPM is the limit it will update a faulty clock. Use kern.timecounter.hardware="HPET". (In reply to Cy Schubert from comment #107) I'm quite sure I did test it before (see Comment 19). will run a test again with 'kern.timecounter.hardware=HPET' and report back. btw, thanks for all time you all are investing here. appreciated (In reply to Vinícius Zavam from comment #108) PS: any thoughts on Why the very same hardware works fine with 12.1-RELEASE? would like to have a better picture of that. (In reply to Cy Schubert from comment #107) 1. changed the sysctl; 2. ran `ntpdate ntp1`; 3. `service ntpd onestart`. after waiting a long time, it kinda looks better indeed. but, as you might see it's still showing a high offset. would it be recommended just to try other timecounters and see what goes - waiting hours to sync? before I waited just a few minutes (<20, maybe) root@head:~ # uptime 4:15PM up 1:41, 2 users, load averages: 0.33, 0.29, 0.26 root@head:~ # ntpq -4 -n -c rl -wp localhost associd=0 status=0615 leap_none, sync_ntp, 1 event, clock_sync, version="ntpd 4.2.8p12-a (1)", processor="amd64", system="FreeBSD/13.0-CURRENT", leap=00, stratum=2, precision=-19, rootdelay=8.046, rootdisp=20.316, refid=131.188.3.223, reftime=e1c07e15.87120556 Wed, Jan 8 2020 16:13:09.527, clock=e1c07e9b.f46a5040 Wed, Jan 8 2020 16:15:23.954, peer=54426, tc=7, mintc=3, offset=12.405128, frequency=-8.399, sys_jitter=0.840451, clk_jitter=4.580, clk_wander=0.114, tai=37, leapsec=201701010000, expire=202006280000 remote refid st t when poll reach delay offset jitter ============================================================================== 0.freebsd.pool.ntp.org .POOL. 16 p - 64 0 0.000 0.000 0.002 +162.159.200.123 10.48.8.4 3 u 124 128 377 1.899 11.758 1.183 -85.114.128.137 131.188.3.220 2 u 57 128 377 2.247 10.270 1.197 -131.188.3.220 .GPS. 1 u 111 128 377 7.906 10.808 1.330 *131.188.3.223 .PZFs. 1 u 1 128 377 8.046 12.701 2.644 +85.220.190.246 213.172.96.14 2 u 55 128 377 4.577 12.635 1.585 (In reply to Vinícius Zavam from comment #110) What does ntptime say? What model of Dell is this? what is the CPU model? (You can get that from dmesg.) Do you have other Dell computers of this same model? What are their results? Use ntpdate or ntpd -G to force a large reset when it boots. (In reply to Cy Schubert from comment #111) I reported the model and hardware specs before: * Comment 20 * Comment 38 * Comment 43 + https://www.dell.com/en-us/work/shop/povw/poweredge-r440 + https://ark.intel.com/content/www/us/en/ark/products/193381/intel-xeon-bronze-3204-processor-8-25m-cache-1-90-ghz.html the '-g' and '-G' parameters where used already to try working the issue around though: * Comment 0 + ntpd_flags="${ntpd_flags} -4 --logfile=/var/log/ntpd.log --statsdir=/etc/ntp/stats -g -g -g -G" the machine running stable/12 ran with the different timecounter (HPET) over night, and here are the results: - I did ran ntpdate before starting ntpd again. ---------- % grep -i ntp /etc/rc.conf ntpdate_enable="YES" ntpd_enable="YES" ntpd_flags="${ntpd_flags} -4 --logfile=/var/log/ntpd.log --statsdir=/etc/ntp/stats" % ps afxuww | grep ntp | head -1 root 17401 0.0 0.1 19004 19084 - Ss 17:09 0:04.86 /usr/sbin/ntpd -4 --logfile=/var/log/ntpd.log --statsdir=/etc/ntp/stats -p /var/db/ntp/ntpd.pid -c /etc/ntp.conf -f /var/db/ntp/ntpd.drift -g % ls -lash /var/db/ntp/ntpd.pid 4 -rw-r--r-- 1 ntpd ntpd 5B Jan 8 17:10 /var/db/ntp/ntpd.pid % ntptime ntp_gettime() returns code 0 (OK) time e1c176f9.27c9cd40 Thu, Jan 9 2020 10:55:05.155, (.155423010), maximum error 506023 us, estimated error 1 us, TAI offset 37 ntp_adjtime() returns code 0 (OK) modes 0x0 (), offset 0.000 us, frequency 439.883 ppm, interval 1 s, maximum error 506023 us, estimated error 1 us, status 0x2001 (PLL,NANO), time constant 6, precision 1.000 us, tolerance 496 ppm, % ntpq -4 -n -c rl -wp 127.0.0.1 associd=0 status=c018 leap_alarm, sync_unspec, 1 event, no_sys_peer, version="ntpd 4.2.8p12-a (1)", processor="amd64", system="FreeBSD/12.1-STABLE", leap=11, stratum=16, precision=-19, rootdelay=0.000, rootdisp=1.080, refid=STEP, reftime=(no time), clock=e1c17704.a4498bc5 Thu, Jan 9 2020 10:55:16.641, peer=0, tc=6, mintc=3, offset=0.000000, frequency=439.883, sys_jitter=0.001907, clk_jitter=0.002, clk_wander=0.389, tai=37, leapsec=201701010000, expire=202006280000 remote refid st t when poll reach delay offset jitter ============================================================================== 10.0.5.3 134.130.4.17 2 u 29 64 1 0.276 -68.196 12.686 10.0.5.4 192.53.103.108 2 u 28 64 1 0.135 -51.048 16.505 10.0.32.248 129.143.2.23 2 u 30 64 1 1.048 -67.666 12.244 10.0.32.249 129.143.2.33 2 u 71 64 1 0.912 -50.333 0.002 % date Thu Jan 9 10:58:23 CET 2020 ---------- the very same hardware (also using HPET), running HEAD reports the following: root@head:~ # ntpq -4 -n -c rl -wp localhost associd=0 status=0615 leap_none, sync_ntp, 1 event, clock_sync, version="ntpd 4.2.8p12-a (1)", processor="amd64", system="FreeBSD/13.0-CURRENT", leap=00, stratum=2, precision=-19, rootdelay=7.920, rootdisp=9.833, refid=131.188.3.220, reftime=e1c1766b.77ca8f6e Thu, Jan 9 2020 9:52:43.467, clock=e1c177fd.00999011 Thu, Jan 9 2020 9:59:25.002, peer=54425, tc=6, mintc=3, offset=-0.189879, frequency=3.565, sys_jitter=0.134864, clk_jitter=0.439, clk_wander=0.010, tai=37, leapsec=201701010000, expire=202006280000 remote refid st t when poll reach delay offset jitter ============================================================================== 0.freebsd.pool.ntp.org .POOL. 16 p - 64 0 0.000 0.000 0.002 +162.159.200.123 10.48.8.4 3 u 23 64 377 1.797 -0.190 3.075 +85.114.128.137 131.188.3.220 2 u 22 64 377 2.190 1.941 0.149 *131.188.3.220 .GPS. 1 u 6 64 377 7.920 -0.505 1.675 +131.188.3.223 .PZFs. 1 u 26 64 377 7.956 -0.441 0.079 +85.220.190.246 213.172.96.14 2 u 3 64 377 4.450 -0.312 2.236 root@head:~ # uptime 9:59AM up 19:26, 2 users, load averages: 0.25, 0.25, 0.24 (In reply to Vinícius Zavam from comment #112) now I do remember it! I reported testing HPET and other timecounters before as a NO FIX for this issue because of that offset we see on the machine running stable/12. 'ACPI-fast' was also used; I did not bother trying 'i8254' or 'dummy' tbh (In reply to Vinícius Zavam from comment #112) forgot to mention that this particular kernel for HEAD is not using 'EARLY_AP_STARTUP' as per Comment 104 (In reply to Vinícius Zavam from comment #112) sharing the ntptime taken from HEAD: root@head:~ # ntptime ntp_gettime() returns code 0 (OK) time e1c17f70.39360d04 Thu, Jan 9 2020 10:31:12.223, (.223481167), maximum error 55129 us, estimated error 270 us, TAI offset 37 ntp_adjtime() returns code 0 (OK) modes 0x0 (), offset -398.303 us, frequency 3.509 ppm, interval 1 s, maximum error 55129 us, estimated error 270 us, status 0x2001 (PLL,NANO), time constant 6, precision 1.000 us, tolerance 496 ppm, just as 'FYI', I built bootable USB images of stable/12@r353572 and stable/12@r352547 to try setting up ntpd for testing and both images are having/presenting the very same timecounter issues. world and kernel (GENERIC) all using their default settings. Just out of interest, how much swap is used? (In reply to Cy Schubert from comment #117) something like nothing; I mean, for an USB "livecd" image (is there any swap to be used?). * quick note: stable/12@r352480 is OK (it served to branch releng/12.1). I am about to test r352513. I think that we now share the opinion that this issue is not directly attached/related to ntpd itself. right? asking just to be sure we are on the same page :) there's something else causing unexpected conditions that do not play well with NTP running on this particular hardware. as reported on Comment 118, I should test r352513 in a few. if it works I will walk til we find a problematic revision/commit. (In reply to Vinícius Zavam from comment #118) so, r352513 worked just fine. it must be something between this rev (that works fine) and r352547 (which do not work as expected). (In reply to Vinícius Zavam from comment #120) r352530 is broken (for this particular case I reported over this PR). so, since r352513 what landed on stable/12 that, IMHO, can be relevant: https://svnweb.freebsd.org/base?view=revision&revision=352517 - https://reviews.freebsd.org/D20793 https://svnweb.freebsd.org/base?view=revision&revision=352519 - https://reviews.freebsd.org/D21027 can we invite someone to check if it really makes sense? The only stable/12 commits between r352513 and r352530 are the two you identified. Can you experiment by removing r352519 and later r352517? Not both at the same time, we need to identify which of the two might be the problem. I also want to look at r352865. It removes the mlockall() for ntp. mlockall() may circumvent your problem. I'm still puzzled why only this one machine has the issue. How many physical cores does this machine have? How many NUMA domains does it report? Created attachment 210689 [details]
svnlite_diff_stable_12__r352513_r352517.patch
Created attachment 210690 [details]
svnlite_diff_stable_12__r352513_r352519.patch
Created attachment 210691 [details]
svnlite_diff_stable_12__r352517_r352519.patch
(In reply to Cy Schubert from comment #122) well, it looks like the puzzle is a bit tricky indeed :) * r352513 works; * r352517 works; * r352519 *WITHOUT* r352517 works fine [1]; * r352519 and further rev do not work. [1] checked r352513 out, and patched with attachment 210691 [details] (https://bz-attachments.freebsd.org/attachment.cgi?id=210691) *ONLY*. how did I generated the diff (with security/ca_root_nss installed)? # svnlite diff https://svn.freebsd.org/base/stable/12@revA https://svn.freebsd.org/base/stable/12@revB > /tmp/patch_name how did I apply the patch to r352513? # svnlite co https://svn.freebsd.org/base/stable/12@r352513 /usr/src # cd /usr/src # svnlite patch /tmp/svnlite_diff_stable_12__r352517_r352519.patch (In reply to Cy Schubert from comment #122) one physical socket, one numa domain (numa-domain 0). I've added alc@ to this PR to get his take on this. Did you also try to remove the mlockall(), r352865, patch to ntpd? r352865 may make ntpd more sensitive to r352517 and r352519. (In reply to Cy Schubert from comment #128) I ran ntpd for a while with `rlimit memlock 32` on its config and it did not help ...but, I used stable/12@r354893 which also has an extra MFC patch related to pmap changes (https://svnweb.freebsd.org/base?view=revision&revision=352541) (In reply to Vinícius Zavam from comment #129) booted back the memstick with r352519 on it and used `rlimit memlock 32` on the ntp.conf: NOT OK. rlimit memlock 32 is not the same as reverting r352865. It occurs after the config file is read while r352865 occurs before. Please try it without r352865. would it be fine if I revert it (contrib/ntp/ntpd/ntpd.c) using the very last rev available under stable/12? (In reply to Vinícius Zavam from comment #132) sorry, I meant: revert these guys to a rev previous to r352865: . contrib/ntp/ntpd/ntpd.c . usr.sbin/ntp/config.h . usr.sbin/ntp/ntpd/ntp.conf Created attachment 210708 [details]
DIFF_base_stable_12_r356668__without__r352865.patch
patch created using svnlite. reverted to previous rev (see diff).
(In reply to Vinícius Zavam from comment #134) Yes, this would test the sensitivity of the mlockall() removal at ntpd startup. If this resolves the issue, the mlockall() removal from ntpd is incompatible with r352517. However I cannot see why this would be the case. (In reply to Cy Schubert from comment #135) it did NOT work; I used r356668 *without* r352865 :'( btw, all memstick images I did test are available here on my side still. if you want that I upload them to any shared server you can send me credentials and I can do it. the machine is now running the 'r356668 *without* r352865' image (using TSC as timecounter). in a few I should write back its stats/drifts/offsets. (In reply to Vinícius Zavam from comment #136) root@freebsd-builder2:~ # svnlite info /usr/src/ Path: /usr/src Working Copy Root Path: /usr/src URL: svn://svn.freebsd.lab.localdomain/base/stable/12 Relative URL: ^/stable/12 Repository Root: svn://svn.freebsd.lab.localdomain/base Repository UUID: ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f Revision: 356668 Node Kind: directory Schedule: normal Last Changed Author: sbruno Last Changed Rev: 356665 Last Changed Date: 2020-01-12 20:19:00 +0000 (Sun, 12 Jan 2020) root@freebsd-builder2:~ # svnlite status /usr/src/ M /usr/src/contrib/ntp/ntpd/ntpd.c M /usr/src/usr.sbin/ntp/config.h M /usr/src/usr.sbin/ntp/ntpd/ntp.conf ---------- *** *** below, the live usb image; *** # uptime 12:28PM up 2:48, 2 users, load averages: 0.17, 0.18, 0.20 ---------- # /usr/sbin/ntpd --logfile=/var/log/ntpd.log --statsdir=/var/db/ntp/stats -p /var/db/ntp/ntpd.pid -c /var/etc_ntp.conf -f /var/db/ntp/ntpd.drift -4 -g ---------- # cat /var/etc_ntp.conf leapfile "/var/db/ntpd.leap-seconds.list" pool 0.freebsd.pool.ntp.org iburst restrict 127.0.0.1 restrict ::1 restrict default limited kod nomodify notrap noquery nopeer restrict source limited kod nomodify notrap noquery statistics clockstats cryptostats loopstats peerstats rawstats sysstats tos minclock 3 maxclock 6 ---------- # ntptime ntp_gettime() returns code 0 (OK) time e1c83294.c1afaf88 Tue, Jan 14 2020 12:29:08.756, (.756587809), maximum error 1218537 us, estimated error 0 us, TAI offset 37 ntp_adjtime() returns code 0 (OK) modes 0x0 (), offset 0.000 us, frequency 500.000 ppm, interval 1 s, maximum error 1218537 us, estimated error 0 us, status 0x2001 (PLL,NANO), time constant 6, precision 1.000 us, tolerance 496 ppm, ---------- # ntpq -n -4 -c rl -wp localhost associd=0 status=c613 leap_alarm, sync_ntp, 1 event, spike_detect, version="ntpd 4.2.8p12-a (1)", processor="amd64", system="FreeBSD/12.1-STABLE", leap=11, stratum=3, precision=-23, rootdelay=14.764, rootdisp=1196.413, refid=159.69.150.81, reftime=e1c83255.24754914 Tue, Jan 14 2020 12:28:05.142, clock=e1c83313.ba52aef1 Tue, Jan 14 2020 12:31:15.727, peer=9746, tc=6, mintc=3, offset=0.000000, frequency=500.000, sys_jitter=178.672719, clk_jitter=0.000, clk_wander=0.000, tai=37, leapsec=201701010000, expire=202006280000 remote refid st t when poll reach delay offset jitter ============================================================================== 0.freebsd.pool.ntp.org .POOL. 16 p - 64 0 0.000 0.000 0.000 +82.64.45.50 .GPS. 1 u 40 64 3 31.154 812.666 299.335 +136.243.177.133 131.188.3.222 2 u 39 64 3 9.073 809.245 302.204 #213.209.109.44 10.129.40.211 2 u 40 64 3 11.941 634.287 433.084 #5.9.121.21 131.188.3.220 2 u 36 64 3 9.066 1392.44 519.254 #85.214.255.221 124.216.164.14 2 u 20 64 3 18.800 636.070 466.311 #138.201.64.208 131.188.3.222 2 u 1 64 7 9.095 644.750 466.129 +78.46.253.198 237.17.204.95 2 u 65 64 3 9.259 876.089 245.915 *159.69.150.81 131.188.3.220 2 u 64 64 3 7.593 877.861 244.791 +90.187.7.5 .DCFa. 1 u 48 64 3 19.764 1019.84 257.518 +178.63.247.119 17.253.54.123 2 u 49 64 3 9.280 905.753 250.617 +213.172.105.106 213.172.96.14 2 u 16 64 7 4.479 841.065 314.702 +51.75.67.47 129.70.132.36 3 u 21 64 3 9.749 1422.92 525.558 ---------- # ls -lash /var/db/ntp/ntpd.drift 4 -rw-r--r-- 1 root ntpd 8B Jan 14 11:25 /var/db/ntp/ntpd.drift ---------- # cat /var/db/ntp/ntpd.drift 500.000 ---------- # cat /var/db/ntp/stats/loopstats 58862 37842.264 0.000000000 500.000 0.000000119 0.000000 6 58862 37851.228 0.114694461 500.000 0.040550616 0.000000 6 58862 38151.836 0.000000000 500.000 0.000000119 0.000000 6 58862 38171.808 0.086842965 500.000 0.030703625 0.000000 6 58862 38206.821 0.092197613 500.000 0.028782938 0.000000 6 58862 38545.516 0.000000000 500.000 0.000000119 0.000000 6 58862 38555.534 0.108082686 500.000 0.038213000 0.000000 6 58862 38606.516 0.122227920 500.000 0.036093146 0.000000 6 58862 38942.190 0.000000000 500.000 0.000000119 0.000000 6 58862 38949.215 0.121292319 500.000 0.042883311 0.000000 6 58862 39257.819 0.000000000 500.000 0.000000119 0.000000 6 58862 39559.249 0.000000000 500.000 0.000000119 0.000000 6 58862 40072.803 0.000000000 500.000 0.000000119 0.000000 6 58862 40502.738 0.000000000 500.000 0.000000119 0.000000 6 58862 40958.617 0.000000000 500.000 0.000000119 0.000000 6 58862 41395.446 0.000000000 500.000 0.000000119 0.000000 6 58862 41806.254 0.000000000 500.000 0.000000119 0.000000 6 58862 42284.875 0.000000000 500.000 0.000000119 0.000000 6 58862 42945.170 0.000000000 500.000 0.000000119 0.000000 6 58862 43514.896 0.000000000 500.000 0.000000119 0.000000 6 58862 44118.361 0.000000000 500.000 0.000000119 0.000000 6 58862 44630.151 0.000000000 500.000 0.000000119 0.000000 6 58862 45208.048 0.000000000 500.000 0.000000119 0.000000 6 # wc /var/db/ntp/stats/peerstats 754 6032 62443 /var/db/ntp/stats/peerstats # head -50 /var/db/ntp/stats/peerstats 58862 37531.533 213.209.109.44 1014 1.077890359 0.012070961 7.937500180 0.000000119 58862 37531.553 82.64.45.50 1014 1.079504466 0.032085015 7.937500315 0.000000119 58862 37532.530 136.243.177.133 1014 1.080806975 0.009437440 7.937500190 0.000000119 58862 37533.556 213.209.109.44 1014 1.082719493 0.012121513 3.937507770 0.004829134 58862 37533.576 82.64.45.50 1014 1.084433245 0.031886184 3.937507971 0.004928778 58862 37534.557 136.243.177.133 1014 1.085543716 0.009227317 3.937507783 0.004736741 58862 37535.526 5.9.121.21 1014 1.087885609 0.009139170 7.937500143 0.000000119 58862 37535.529 213.209.109.44 1014 1.087242752 0.011981241 1.937515314 0.007345990 58862 37535.549 82.64.45.50 1014 1.088564439 0.032337657 1.937515553 0.007040947 58862 37536.535 136.243.177.133 1014 1.090188088 0.009078699 1.937515329 0.007401874 58862 37537.506 5.9.121.21 1014 1.092506565 0.009234921 3.937507715 0.004620956 58862 37537.509 213.209.109.44 1014 1.091955616 0.011985290 0.937520962 0.010088748 58862 37539.651 213.209.109.44 1014 0.002468651 0.012097263 7.937500180 0.000000119 58862 37541.649 5.9.121.21 1024 0.007802739 0.009111893 7.937500143 0.000000119 58862 37541.652 213.209.109.44 1014 0.007193125 0.011922935 3.937507769 0.004724473 58862 37543.649 5.9.121.21 1024 0.012528231 0.009260437 3.937507715 0.004725492 58862 37543.649 136.243.177.133 1024 0.012420368 0.009333823 7.937500189 0.000000119 58862 37543.652 213.209.109.44 1014 0.011880933 0.011957675 1.937515314 0.007435274 58862 37545.642 5.9.121.21 1024 0.017204421 0.009149525 1.937515251 0.007424903 58862 37545.642 136.243.177.133 1024 0.017034418 0.009224962 3.937507783 0.004614050 58862 37545.645 213.209.109.44 1014 0.016577549 0.011952335 0.937520961 0.010151956 58862 37547.612 5.9.121.21 1024 0.021821418 0.009220313 0.937520894 0.010069812 58862 37547.612 136.243.177.133 1024 0.021800885 0.009191850 1.937515330 0.007440205 58862 37547.615 213.209.109.44 161a 0.021225211 0.011961550 0.437524722 0.012822093 58862 37549.609 5.9.121.21 1424 0.026448697 0.009266346 0.437524654 0.012731059 58862 37549.610 136.243.177.133 1024 0.026397545 0.009227886 0.937520978 0.010069076 58862 37551.640 5.9.121.21 161a 0.031312830 0.009092092 0.187527001 0.015610121 58862 37551.640 136.243.177.133 1424 0.031181775 0.009229372 0.437524740 0.012874665 58862 37553.635 136.243.177.133 1424 0.035877015 0.009100215 0.187527089 0.015590387 58862 37580.644 213.209.109.44 141a 0.098862243 0.012097064 0.187752307 0.087245607 58862 37607.600 5.9.121.21 141a 0.162869339 0.009261819 0.062927082 0.143572624 58862 37607.623 82.64.45.50 1024 0.163679980 0.032040971 7.937500315 0.000000119 58862 37609.612 82.64.45.50 1024 0.168943986 0.031597416 3.937507969 0.005264006 58862 37611.672 82.64.45.50 1024 0.173166858 0.032193042 1.937515550 0.007342802 58862 37613.620 82.64.45.50 1024 0.178066571 0.031590011 0.937521212 0.010233974 58862 37615.634 82.64.45.50 1024 0.182960282 0.031458396 0.437524979 0.013115317 58862 37617.623 82.64.45.50 1024 0.187942028 0.032190200 0.187527337 0.016063752 58862 37645.649 136.243.177.133 131a 0.252399282 0.009334014 0.063192952 0.228421388 58862 37676.592 5.9.121.21 162a 0.325252766 0.009265608 0.000727142 0.289726091 58862 37706.652 213.209.109.44 142a 0.395491959 0.012302270 0.063556569 0.370594487 58862 37706.672 82.64.45.50 141a 0.396911066 0.031591768 0.063171050 0.221270238 58862 37711.618 136.243.177.133 142a 0.407560780 0.009317224 0.000837798 0.359886482 58862 37738.634 85.214.255.221 1014 0.468418039 0.019533679 7.937500266 0.000000119 58862 37740.659 85.214.255.221 1014 0.472941001 0.019237409 3.937507896 0.004522962 58862 37742.659 85.214.255.221 1014 0.477717306 0.019382327 1.937515463 0.007392208 58862 37744.647 5.9.121.21 143a 0.490381900 0.019403816 0.000865851 0.419574702 58862 37744.647 85.214.255.221 1014 0.482478367 0.019473098 0.937521122 0.010186951 58862 37746.651 85.214.255.221 1014 0.487110119 0.019389783 0.437524888 0.012843635 58862 37772.617 82.64.45.50 162a 0.551768500 0.032194044 0.000826974 0.353053955 58862 37774.600 213.209.109.44 132a 0.555208404 0.012542652 0.001034484 0.494262673 # tail -50 /var/db/ntp/stats/peerstats 58862 44633.151 90.187.7.5 1024 0.637537062 0.021736366 7.937500252 0.000000119 58862 44634.165 51.75.67.47 1034 0.645345351 0.009968341 7.937500149 0.000000119 58862 44635.163 159.69.150.81 1034 0.644559836 0.007822520 7.937500148 0.000000119 58862 44635.164 138.201.64.208 1064 0.644749522 0.009094834 7.937500158 0.000000119 58862 44698.130 78.46.253.198 1014 0.761280623 0.009279784 3.937751802 0.123958674 58862 44700.164 78.46.253.198 1014 0.765024059 0.009266539 1.937637518 0.090337817 58862 44701.162 159.69.150.81 1034 0.766751084 0.007680146 1.937751508 0.086442558 58862 44714.164 178.63.247.119 1024 0.791085693 0.009409210 3.937807718 0.151911016 58862 44723.186 82.64.45.50 1014 0.808524528 0.031606453 3.937841718 0.168421440 58862 44724.112 136.243.177.133 1034 0.809245300 0.009073175 3.937849031 0.172072828 58862 44725.186 82.64.45.50 1014 0.812665961 0.031154007 1.937682417 0.122055513 58862 44729.109 5.9.121.21 1074 0.818480408 0.009120925 1.937864001 0.125693219 58862 44741.137 213.172.105.106 1014 0.841064860 0.004479430 0.938086069 0.149301374 58862 44742.120 51.75.67.47 1034 0.845352059 0.009852784 3.937905223 0.200006708 58862 44744.104 51.75.67.47 1034 0.849026435 0.009851743 1.937714010 0.144047709 58862 44760.134 78.46.253.198 1014 0.876089069 0.009258964 0.937962877 0.165857523 58862 44761.162 159.69.150.81 1034 0.877860560 0.007592801 0.938019651 0.163261743 58862 44776.123 178.63.247.119 1024 0.905753150 0.009280153 0.937980271 0.179391025 58862 44804.165 51.75.67.47 1034 0.960172225 0.009828527 0.938000903 0.203840638 58862 44839.115 90.187.7.5 1024 1.019841188 0.019763973 0.438561726 0.257302837 58862 44913.164 5.9.121.21 1074 1.159282585 0.009118737 0.188933250 0.337521041 58862 45039.102 5.9.121.21 1574 1.392443344 0.009065561 0.064147002 0.519254147 58862 45054.134 51.75.67.47 1434 1.422925249 0.009749120 0.064567247 0.525558154 58862 45121.125 51.75.67.47 1434 1.547027949 0.009718224 0.001532345 0.597976490 58862 45127.139 213.172.105.106 141a 1.556211612 0.004437814 0.002580784 0.604643715 58862 45141.130 138.201.64.208 1564 0.752094280 0.009113566 0.005565666 0.493406740 58862 45145.145 78.46.253.198 141a 1.588989862 0.009257890 0.002427393 0.602722418 58862 45169.116 213.209.109.44 1534 1.382539646 0.012003556 0.003461257 0.401974337 58862 45188.129 51.75.67.47 1434 1.671012142 0.009708984 0.001228419 0.600794495 58862 45190.162 85.214.255.221 1514 1.188130962 0.019081293 0.003548610 0.320168745 58862 45207.107 138.201.64.208 1564 0.866947303 0.009163973 0.005393284 0.514416030 58862 45209.105 78.46.253.198 1014 0.764071175 0.009364904 7.937500368 0.000000119 58862 45209.108 213.209.109.44 1044 0.761240965 0.012549930 7.937500184 0.000000119 58862 45210.106 138.201.64.208 1074 0.765904433 0.009310517 7.937500159 0.000000119 58862 45210.116 85.214.255.221 1024 0.765277075 0.019859849 7.937500268 0.000000119 58862 45210.120 90.187.7.5 1014 0.761396771 0.024153187 7.937500271 0.000000119 58862 45210.128 82.64.45.50 1024 0.766820991 0.032057241 7.937500315 0.000000119 58862 45211.066 51.75.67.47 1044 0.770099725 0.009669124 7.937500147 0.000000119 58862 45212.069 213.172.105.106 1014 0.769277463 0.004626638 7.937500571 0.000000119 58862 45212.074 5.9.121.21 1084 0.769345653 0.009210194 7.937500144 0.000000119 58862 45212.074 178.63.247.119 1034 0.769607200 0.009594431 7.937500146 0.000000119 58862 45213.072 136.243.177.133 1044 0.771148616 0.009084254 7.937500187 0.000000119 58862 45213.084 159.69.150.81 1014 0.774089209 0.013293912 7.937500189 0.000000119 58862 45258.047 159.69.150.81 1014 0.854471430 0.007752637 3.937668992 0.080382221 58862 45260.060 159.69.150.81 1014 0.858243844 0.007731180 1.937595893 0.059566071 58862 45274.058 90.187.7.5 1014 0.879748458 0.023653975 3.937740402 0.118351687 58862 45275.104 178.63.247.119 1034 0.886369234 0.009448902 1.937740255 0.082609415 58862 45284.063 213.209.109.44 1044 0.900025164 0.012018858 1.937785338 0.098358518 58862 45284.083 82.64.45.50 1024 0.903789294 0.031609554 1.937781808 0.096963290 58862 45286.085 5.9.121.21 1084 0.906434434 0.009101309 3.937777715 0.137088781 # cat /var/db/ntp/stats/sysstats 58862 41134.563 3600 743 718 731 0 0 0 0 0 0 0 58862 44741.133 3600 694 694 694 0 0 0 0 0 0 0 # wc /var/db/ntp/stats/rawstats 3162 53902 577044 /var/db/ntp/stats/rawstats # head -50 /var/db/ntp/stats/rawstats 58862 37528.567 10.0.86.163 82.64.45.50 0.000000000 0.000000000 2551564513.473209355 0.000000000 3 4 3 0 6 -23 0.000000 0.000000 .INIT. 58862 37529.500 10.0.86.163 136.243.177.133 0.000000000 0.000000000 2568341729.883757025 0.000000000 3 4 3 0 6 -23 0.000000 16777216.000000 .INIT. 58862 37530.508 10.0.86.163 213.209.109.44 0.000000000 0.000000000 2585118945.699312538 0.000000000 3 4 3 0 6 -23 0.000000 33554432.000000 .INIT. 58862 37531.521 10.0.86.163 213.209.109.44 0.000000000 0.000000000 2601896161.646427484 0.000000000 3 4 3 0 6 -23 0.000000 50331648.000000 .INIT. 58862 37531.521 10.0.86.163 82.64.45.50 0.000000000 0.000000000 2601896161.040211291 0.000000000 3 4 3 0 6 -23 0.000000 50331648.000000 .INIT. 58862 37531.521 10.0.86.163 5.9.121.21 0.000000000 0.000000000 2601896161.317005904 0.000000000 3 4 3 0 6 -23 0.000000 50331648.000000 .INIT. 58862 37531.533 213.209.109.44 10.0.86.163 3787986331.520591536 3787986332.604517375 3787986332.604539781 3787986331.532684902 0 4 4 2 6 -24 0.000290 0.024048 10.129.40.211 58862 37531.553 82.64.45.50 10.0.86.163 3787986331.520649614 3787986332.616196588 3787986332.616201415 3787986331.552739456 0 4 4 1 6 -25 0.000000 0.000000 .GPS. 58862 37532.520 10.0.86.163 136.243.177.133 0.000000000 0.000000000 2618673377.145006509 0.000000000 3 4 3 0 6 -23 0.000000 67108864.000000 .INIT. 58862 37532.530 136.243.177.133 10.0.86.163 3787986332.520097682 3787986333.605623377 3787986333.605792078 3787986332.529703823 0 4 4 2 6 -23 0.008667 0.021744 131.188.3.222 58862 37533.544 10.0.86.163 213.209.109.44 2618673377.488750672 2601896161.664205464 2635450593.370243641 0.000000000 3 4 3 0 6 -23 0.000000 83886080.000000 .INIT. 58862 37533.544 10.0.86.163 82.64.45.50 2618673377.239757515 2601896161.247398409 2635450593.836476180 0.000000000 3 4 3 0 6 -23 0.000000 83886080.000000 .INIT. 58862 37533.556 213.209.109.44 10.0.86.163 3787986333.544094584 3787986334.632874834 3787986334.632945914 3787986333.556287177 0 4 4 2 6 -24 0.000290 0.024078 10.129.40.211 58862 37533.576 82.64.45.50 10.0.86.163 3787986333.544145813 3787986334.644522149 3787986334.644526978 3787986333.576036826 0 4 4 1 6 -25 0.000000 0.000000 .GPS. 58862 37534.548 10.0.86.163 136.243.177.133 2635450593.551514960 2618673377.463555963 2652227809.803303394 0.000000000 3 4 3 0 6 -23 0.000000 100663296.000000 .INIT. 58862 37534.557 136.243.177.133 10.0.86.163 3787986334.548014033 3787986335.638171407 3787986335.638243568 3787986334.557313510 0 4 4 2 6 -23 0.008667 0.021774 131.188.3.222 58862 37535.517 10.0.86.163 5.9.121.21 0.000000000 0.000000000 2669005025.568073184 0.000000000 3 4 3 0 6 -23 0.000000 117440512.000000 .INIT. 58862 37535.517 10.0.86.163 213.209.109.44 2652227809.311493434 2635450593.136084113 2669005025.115940125 0.000000000 3 4 3 0 6 -23 0.000000 117440512.000000 .INIT. 58862 37535.517 10.0.86.163 82.64.45.50 2652227809.334854105 2635450593.246680711 2669005025.481388182 0.000000000 3 4 3 0 6 -23 0.000000 117440512.000000 .INIT. 58862 37535.526 5.9.121.21 10.0.86.163 3787986335.516577576 3787986336.609032770 3787986336.609081100 3787986335.525765076 0 4 4 2 6 -25 0.008545 0.029251 131.188.3.220 58862 37535.529 213.209.109.44 10.0.86.163 3787986335.516611940 3787986336.609845313 3787986336.609861169 3787986335.528609037 0 4 4 2 6 -24 0.000290 0.024109 10.129.40.211 58862 37535.549 82.64.45.50 10.0.86.163 3787986335.516620426 3787986336.621353694 3787986336.621358517 3787986335.548962906 0 4 4 1 6 -25 0.000000 0.000000 .GPS. 58862 37536.526 10.0.86.163 136.243.177.133 2669005025.206762531 2652227809.141016755 2685782241.348000319 0.000000000 3 4 3 0 6 -23 0.000000 134217728.000000 .INIT. 58862 37536.535 136.243.177.133 10.0.86.163 3787986336.525575062 3787986337.620302500 3787986337.620373700 3787986336.534724962 0 4 4 2 6 -23 0.008667 0.021805 131.188.3.222 58862 37537.497 10.0.86.163 5.9.121.21 2685782241.186491764 2669005025.244302304 2702559457.688707381 0.000000000 3 4 3 0 6 -23 0.000000 150994944.000000 .INIT. 58862 37537.497 10.0.86.163 213.209.109.44 2685782241.569765068 2669005025.003605993 2702559457.660066634 0.000000000 3 4 3 0 6 -23 0.000000 150994944.000000 .INIT. 58862 37537.497 10.0.86.163 82.64.45.50 2685782241.052155591 2669005025.261070761 2702559457.192552715 0.000000000 3 4 3 0 6 -23 0.000000 150994944.000000 .INIT. 58862 37537.506 5.9.121.21 10.0.86.163 3787986337.496586781 3787986338.593710806 3787986338.593800735 3787986337.505911630 0 4 4 2 6 -25 0.008545 0.029282 131.188.3.220 58862 37537.509 213.209.109.44 10.0.86.163 3787986337.496596972 3787986338.594545233 3787986338.594569266 3787986337.508606294 0 4 4 2 6 -24 0.000290 0.024139 10.129.40.211 58862 37539.639 10.0.86.163 213.209.109.44 0.000000000 0.000000000 2736113889.033362784 0.000000000 3 4 3 0 6 -23 0.000000 0.000000 .STEP. 58862 37539.651 213.209.109.44 10.0.86.163 3787986339.638527514 3787986339.647044797 3787986339.647099708 3787986339.650679689 0 4 4 2 6 -24 0.000290 0.024155 10.129.40.211 58862 37541.640 10.0.86.163 5.9.121.21 0.000000000 0.000000000 2769668321.710583963 0.000000000 3 4 3 0 6 -23 0.000000 33554432.000000 .STEP. 58862 37541.640 10.0.86.163 213.209.109.44 2736113889.583307781 2736113889.687279859 2769668321.749722936 0.000000000 3 4 3 0 6 -23 0.000000 33554432.000000 .STEP. 58862 37541.649 5.9.121.21 10.0.86.163 3787986341.639967484 3787986341.652326169 3787986341.652411091 3787986341.649164299 0 4 4 2 6 -25 0.008545 0.029327 131.188.3.220 58862 37541.652 213.209.109.44 10.0.86.163 3787986341.640013560 3787986341.653168153 3787986341.653238435 3787986341.652006778 0 4 4 2 6 -24 0.000290 0.024185 10.129.40.211 58862 37543.640 10.0.86.163 5.9.121.21 2769668321.794571200 2769668321.658709505 2803222753.789960244 0.000000000 3 4 3 0 6 -23 0.000000 67108864.000000 .STEP. 58862 37543.640 10.0.86.163 213.209.109.44 2769668321.326694170 2769668321.546553233 2803222753.487363257 0.000000000 3 4 3 0 6 -23 0.000000 67108864.000000 .STEP. 58862 37543.640 10.0.86.163 136.243.177.133 0.000000000 0.000000000 2803222753.153958776 0.000000000 3 4 3 0 6 -23 0.000000 67108864.000000 .STEP. 58862 37543.649 5.9.121.21 10.0.86.163 3787986343.639972376 3787986343.657130826 3787986343.657199790 3787986343.649301777 0 4 4 2 6 -25 0.008545 0.029358 131.188.3.220 58862 37543.649 136.243.177.133 10.0.86.163 3787986343.640005657 3787986343.657092937 3787986343.657199758 3787986343.649446302 0 4 4 2 6 -23 0.008667 0.021896 131.188.3.222 58862 37543.652 213.209.109.44 10.0.86.163 3787986343.639995783 3787986343.657855553 3787986343.657899995 3787986343.651997900 0 4 4 2 6 -24 0.000290 0.024216 10.129.40.211 58862 37545.633 10.0.86.163 5.9.121.21 2803222753.829074780 2803222753.166568318 2836777185.207337298 0.000000000 3 4 3 0 6 -23 0.000000 100663296.000000 .STEP. 58862 37545.633 10.0.86.163 213.209.109.44 2803222753.320837775 2803222753.595060924 2836777185.041062393 0.000000000 3 4 3 0 6 -23 0.000000 100663296.000000 .STEP. 58862 37545.633 10.0.86.163 136.243.177.133 2803222753.301731030 2803222753.891056219 2836777185.709427931 0.000000000 3 4 3 0 6 -23 0.000000 100663296.000000 .STEP. 58862 37545.642 5.9.121.21 10.0.86.163 3787986345.633027327 3787986345.654806511 3787986345.654857640 3787986345.642227981 0 4 4 2 6 -25 0.008545 0.029388 131.188.3.220 58862 37545.642 136.243.177.133 10.0.86.163 3787986345.633081299 3787986345.654728199 3787986345.654799398 3787986345.642377461 0 4 4 2 6 -23 0.008667 0.021927 131.188.3.222 58862 37545.645 213.209.109.44 10.0.86.163 3787986345.633064451 3787986345.655618167 3787986345.655673794 3787986345.645072413 0 4 4 2 6 -24 0.000290 0.024246 10.129.40.211 58862 37547.603 10.0.86.163 5.9.121.21 2836777185.073252002 2836777185.566610911 2870331617.050434268 0.000000000 3 4 3 3 6 -23 570621952.000000 1996095488.000000 213.209.109.44 58862 37547.603 10.0.86.163 213.209.109.44 2836777185.875928560 2836777185.208849170 2870331617.982593811 0.000000000 3 4 3 3 6 -23 570621952.000000 1996095488.000000 213.209.109.44 58862 37547.603 10.0.86.163 136.243.177.133 2836777185.941141167 2836777185.417380490 2870331617.854221678 0.000000000 3 4 3 3 6 -23 570621952.000000 1996095488.000000 213.209.109.44 ... 58862 45075.728 127.0.0.1 127.0.0.1 775168100.438357246 1630351921.160906322 1881156642.396048513 0.000000000 6 2 3 130 0 1 5062.000000 2516647936.000000 118.101.114.115 372: 73736f723d22616d643634222c0d0a73797374656d3d22467265654253442f31 322e312d535441424c45222c206c6561703d332c207374726174756d3d332c20707265636973696f6e3d2d32332c0d0a726f6f7464656c61793d31342e3736342c20726f6f74646973703d313139362e3431332c2072656669643d3135392e36392e3135302e38312c0d0a72656674696d653d307865316 338333235352e32343735343931342c20636c6f636b3d307865316338333331332e62613532616566312c20706565723d393734362c2074633d362c0d0a6d696e74633d332c206f66667365743d302e3030303030302c206672657175656e63793d3530302e3030302c207379735f6a69747465723d3137 382e3637323731392c0d0a636c6b5f6a69747465723d302e3030302c20636c6b5f77616e6465723d302e3030302c207461693d33372c206c6561707365633d3230313730313031303030302c0d0a6578706972653d3230323030363238303030300d0a0000 58862 45075.730 127.0.0.1 127.0.0.1 605295398.101899275 437522726.390946397 336924454.453446278 0.000000000 6 2 3 129 0 2 5062.000000 872415232.000000 38.22.20.52 16: 260d1534260c1434260b1414260a8811 58862 45075.731 127.0.0.1 127.0.0.1 539766830.439017442 1031041647.449709665 1869112178.133750227 0.000000000 6 2 3 162 0 3 170267016.000000 3556835328.000000 115.114.99.97 432: 302e667265656273642e706f6f6c2e6e74702e6f7267222c0d0a647374616 4723d302e302e302e302c20647374706f72743d302c206c6561703d332c207374726174756d3d31362c20707265636973696f6e3d2d32332c0d0a726f6f7464656c61793d302e3030302c20726f6f74646973703d302e3030302c2072656669643d504f4f4c2c0d0a72656674696d653d30783030303030 3030302e30303030303030302c207265633d307830303030303030302e30303030303030302c2072656163683d3078302c0d0a756e72656163683d302c20686d6f64653d332c20706d6f64653d302c2068706f6c6c3d362c2070706f6c6c3d31302c20686561647761793d302c20666c6173683d3078313 630302c0d0a6b657969643d302c206f66667365743d302e3030302c2064656c61793d302e3030302c2064697370657273696f6e3d31363030302e3030302c206a69747465723d302e3030302c0d0a66696c7464656c61793d20302e303020302e303020302e303020302e303020302e303020302e303020 302e303020302e30302c0d0a66696c746f66667365743d20302e303020302e303020302e303020302e3030 58862 45075.731 127.0.0.1 127.0.0.1 540028974.188235175 1711934764.392401481 1030779753.188326903 0.000000000 6 2 3 130 0 3 170267016.000000 1778439169.000000 32.48.46.48 72: 30302e30302031363030302e30302031363030302e30302031363030302e3030 2031363030302e30302031363030302e30302031363030302e30302031363030302e30300d0a0000 58862 45075.735 127.0.0.1 127.0.0.1 775238702.125674260 1885565555.240058090 741552689.454885729 0.000000000 6 2 3 162 0 4 187044884.000000 3556835328.000000 115.114.99.97 432: 6164723d31302e302e38362e3136332c20647374706f72743d3132332c0d0a 6c6561703d302c207374726174756d3d312c20707265636973696f6e3d2d32352c20726f6f7464656c61793d302e3030302c20726f6f74646973703d302e3030302c0d0a72656669643d4750532c2072656674696d653d307865316338333265632e30303030303030302c207265633d307865316338333 265622e32623639623734622c0d0a72656163683d3078332c20756e72656163683d302c20686d6f64653d332c20706d6f64653d342c2068706f6c6c3d362c2070706f6c6c3d362c20686561647761793d3436382c0d0a666c6173683d3078302c206b657969643d302c206f66667365743d3831322e3636 362c2064656c61793d33312e3135342c2064697370657273696f6e3d36362e3239322c0d0a6a69747465723d3239392e3333352c20786c656176653d302e3032302c0d0a66696c7464656c61793d2033322e35302033312e33312033312e37362033312e34362033312e31352033312e36312033312e373 620302e30302c0d0a66696c746f66667365743d20313338352e383620313135332e313820313033382e35 58862 45075.736 127.0.0.1 127.0.0.1 540489262.180544866 908079669.192111027 774905904.051455509 0.000000000 6 2 3 130 0 4 187044884.000000 1543558145.000000 48.32.57.50 56: 0a66696c74646973703d20302e303020312e383920322e383220332e373520342e 363520342e363820362e30352031363030302e30300d0a 58862 45075.740 127.0.0.1 127.0.0.1 825111348.192111445 539767603.439017442 1031041647.172656190 0.000000000 6 2 3 162 0 5 203830292.000000 3556835328.000000 115.114.99.97 432: 206473746164723d31302e302e38362e3136332c20647374706f72743d3132 332c0d0a6c6561703d302c207374726174756d3d322c20707265636973696f6e3d2d32332c20726f6f7464656c61793d382e3735392c20726f6f74646973703d32332e3830342c0d0a72656669643d3133312e3138382e332e3232322c2072656674696d653d307865316338333164362e6564613330353 7312c0d0a7265633d307865316338333265632e31656665336530392c2072656163683d3078332c20756e72656163683d302c20686d6f64653d332c20706d6f64653d342c0d0a68706f6c6c3d362c2070706f6c6c3d362c20686561647761793d3436392c20666c6173683d3078302c206b657969643d30 2c206f66667365743d3830392e3234352c0d0a64656c61793d392e3037332c2064697370657273696f6e3d36372e3232322c206a69747465723d3330322e3230342c20786c656176653d302e3032352c0d0a66696c7464656c61793d20392e333420392e323620392e333220392e323020392e323020392 e303720392e323020302e30302c0d0a66696c746f66667365743d20313338362e393620313135332e3639 58862 45075.740 127.0.0.1 127.0.0.1 540225582.180468632 941634864.196017813 859185205.215594185 0.000000000 6 2 3 130 0 5 203830292.000000 1660998657.000000 32.49.48.51 64: 20302e30302c0d0a66696c74646973703d20302e303020312e383920322e383220 332e373520342e363520342e363820362e30382031363030302e30300d0a00 58862 45075.746 127.0.0.1 127.0.0.1 825112880.203830313 1931488308.435308662 826111090.125674438 0.000000000 6 2 3 162 0 6 220607509.000000 3556835328.000000 115.114.99.97 432: 6473746164723d31302e302e38362e3136332c20647374706f72743d313233 2c0d0a6c6561703d302c207374726174756d3d322c20707265636973696f6e3d2d32342c20726f6f7464656c61793d302e3330352c20726f6f74646973703d382e3230392c0d0a72656669643d31302e3132392e34302e3231312c2072656674696d653d307865316338333264392e62336631623638312 c0d0a7265633d307865316338333265622e32363239363962362c2072656163683d3078332c20756e72656163683d302c20686d6f64653d332c20706d6f64653d342c0d0a68706f6c6c3d362c2070706f6c6c3d362c20686561647761793d3436382c20666c6173683d3078302c206b657969643d302c20 6f66667365743d3633342e3238372c0d0a64656c61793d31312e3934312c2064697370657273696f6e3d36362e3237302c206a69747465723d3433332e3038342c20786c656176653d302e3034372c0d0a66696c7464656c61793d2031322e30302031322e30312031322e33302031322e32392031322e3 2342031322e32392031312e393420302e30302c0d0a66696c746f66667365743d20313338322e35342031 58862 45075.746 127.0.0.1 127.0.0.1 958410801.203830313 808984633.188311471 875575328.125812005 0.000000000 6 2 3 130 0 6 220607509.000000 1761661953.000000 49.52.56.46 72: 3633342e323920302e30302c0d0a66696c74646973703d20302e303020312e3839 20322e383220332e373520342e363520342e363820362e30362031363030302e30300d0a000000 58862 45075.752 127.0.0.1 127.0.0.1 841888050.449709665 1869636466.192343977 539767602.380683147 0.000000000 6 2 3 162 0 7 237401109.000000 3556835328.000000 115.114.99.97 432: 64723d31302e302e38362e3136332c20647374706f72743d3132332c206c65 61703d302c0d0a7374726174756d3d322c20707265636973696f6e3d2d32352c20726f6f7464656c61793d382e3439392c20726f6f74646973703d34372e3037332c0d0a72656669643d3133312e3138382e332e3232302c2072656674696d653d307865316338326166312e30353566666364622c0d0a7 265633d307865316338333265662e31613261656165642c2072656163683d3078332c20756e72656163683d312c20686d6f64653d332c20706d6f64653d342c0d0a68706f6c6c3d362c2070706f6c6c3d362c20686561647761793d3437322c20666c6173683d3078302c206b657969643d302c206f6666 7365743d313339322e3434332c0d0a64656c61793d392e3036362c2064697370657273696f6e3d36342e3134372c206a69747465723d3531392e3235342c20786c656176653d302e3032372c0d0a66696c7464656c61793d20392e303720392e313220392e323220392e323120392e313220392e3232203 92e323020302e30302c0d0a66696c746f66667365743d20313339322e343420313135392e323820313034 58862 45075.752 127.0.0.1 127.0.0.1 775434552.219241631 925774897.203950894 909585968.188204773 0.000000000 6 2 3 130 0 7 237401109.000000 1593889793.000000 52.46.53.53 60: 302c0d0a66696c74646973703d20302e303020312e383920322e383220332e3735 20342e363520342e363820362e30392031363030302e30300d0a00 58862 45075.758 127.0.0.1 127.0.0.1 892481076.196078194 1931488305.435308662 826111090.125674438 0.000000000 6 2 3 162 0 8 254153749.000000 3556835328.000000 115.114.99.97 432: 6473746164723d31302e302e38362e3136332c20647374706f72743d313233 2c0d0a6c6561703d302c207374726174756d3d322c20707265636973696f6e3d2d32332c20726f6f7464656c61793d31312e3936332c20726f6f74646973703d34322e3438302c0d0a72656669643d3132342e3231362e3136342e31342c2072656674696d653d307865316338326432382e30326136633 166342c0d0a7265633d307865316338333266662e32313563613630392c2072656163683d3078332c20756e72656163683d312c20686d6f64653d332c20706d6f64653d342c0d0a68706f6c6c3d362c2070706f6c6c3d362c20686561647761793d3438382c20666c6173683d3078302c206b657969643d 302c206f66667365743d3633362e3037302c0d0a64656c61793d31382e3830302c2064697370657273696f6e3d36372e3030322c206a69747465723d3436362e3331312c20786c656176653d302e3032322c0d0a66696c7464656c61793d2031392e35372031392e30382031392e34332031392e3535203 1392e35352031392e35322031382e383020302e30302c0d0a66696c746f66667365743d20313432312e34 58862 45075.758 127.0.0.1 127.0.0.1 858665783.207902921 808857144.215640552 540292398.180468811 0.000000000 6 2 3 130 0 8 254153749.000000 1811993601.000000 49.32.49.49 72: 3837203633362e303720302e30302c0d0a66696c74646973703d20302e30302031 2e383920322e383220332e373520342e363520342e363820362e33362031363030302e30300d0a 58862 45075.764 127.0.0.1 127.0.0.1 908996912.188265693 1931488312.435308662 826111090.125674438 0.000000000 6 2 3 162 0 9 270951445.000000 3556835328.000000 115.114.99.97 432: 6473746164723d31302e302e38362e3136332c20647374706f72743d313233 2c0d0a6c6561703d302c207374726174756d3d322c20707265636973696f6e3d2d32342c20726f6f7464656c61793d382e3433382c20726f6f74646973703d32372e3534322c0d0a72656669643d3133312e3138382e332e3232322c2072656674696d653d307865316338333030352e636465323636396 52c0d0a7265633d307865316338333331322e32303339626131362c2072656163683d3078372c20756e72656163683d312c20686d6f64653d332c20706d6f64653d342c0d0a68706f6c6c3d362c2070706f6c6c3d362c20686561647761793d3530302c20666c6173683d3078302c206b657969643d302c 206f66667365743d3634342e3735302c0d0a64656c61793d392e3039352c2064697370657273696f6e3d352e3630342c206a69747465723d3436362e3132392c20786c656176653d302e3032342c0d0a66696c7464656c61793d20392e323120392e313820392e323820392e323220392e313620392e313 920392e313120392e30392c0d0a66696c746f66667365743d20313435372e353520313332392e39302031 58862 45075.764 127.0.0.1 127.0.0.1 909647926.207903755 892679968.125842582 775042359.211429190 0.000000000 6 2 3 130 0 9 270951445.000000 1593889793.000000 48.57.54.46 60: 34342e37352c0d0a66696c74646973703d20302e303020312e303420322e393320 332e383620342e373920352e363920352e373220362e35390d0a00 58862 45075.770 127.0.0.1 127.0.0.1 859124270.219622682 1920147500.447012924 842087796.391115916 0.000000000 6 2 3 162 0 10 287709716.000000 3556835328.000000 115.114.99.97 432: 73746164723d31302e302e38362e3136332c20647374706f72743d3132332 c0d0a6c6561703d302c207374726174756d3d322c20707265636973696f6e3d2d32312c20726f6f7464656c61793d31312e3631322c20726f6f74646973703d33332e3930352c0d0a72656669643d3233372e31372e3230342e39352c2072656674696d653d307865316338326638372e64306538623236 382c0d0a7265633d307865316338333264322e32396632306330632c2072656163683d3078332c20756e72656163683d302c20686d6f64653d332c20706d6f64653d342c0d0a68706f6c6c3d362c2070706f6c6c3d362c20686561647761793d3434332c20666c6173683d3078302c206b657969643d302 c206f66667365743d3837362e3038392c0d0a64656c61793d392e3235392c2064697370657273696f6e3d36352e3431392c206a69747465723d3234352e3931352c20786c656176653d302e3034392c0d0a66696c7464656c61793d20392e323620392e323820392e333820392e323620392e323720392e 323820392e333120302e30302c0d0a66696c746f66667365743d20313333392e313220313130352e373520 58862 45075.770 127.0.0.1 127.0.0.1 909582393.196047676 825636640.125857483 775369526.187991273 0.000000000 6 2 3 130 0 10 287709716.000000 1627444225.000000 57.57.48.46 64: 2e30302c0d0a66696c74646973703d20302e303020312e383920322e383220332 e373520342e363520342e363820352e36392031363030302e30300d0a000000 ... # tail -50 /var/db/ntp/stats/rawstats 58862 45285.096 10.0.86.163 136.243.177.133 3811821793.283310350 3811821793.975198510 3845376225.487924343 0.000000000 3 4 3 0 6 -23 0.000000 1258291200.000000 .STEP. 58862 45285.105 136.243.177.133 10.0.86.163 3787994085.096144228 3787994086.005257523 3787994086.005309717 3787994085.105291032 0 4 4 2 6 -23 0.008759 0.027527 131.188.3.222 58862 45286.076 10.0.86.163 5.9.121.21 2620639457.526025226 2620639457.045422558 3862153441.643134896 0.000000000 3 4 3 0 6 -23 0.000000 1275068416.000000 .STEP. 58862 45286.085 5.9.121.21 10.0.86.163 3787994086.076135912 3787994086.987121001 3787994086.987170904 3787994086.085287125 0 4 4 2 6 -25 0.008514 0.023178 131.188.3.220 58862 45288.091 10.0.86.163 5.9.121.21 3862153441.493098735 3862153441.521008794 3895707873.525394743 0.000000000 3 4 3 0 6 -23 0.000000 1308622848.000000 .STEP. 58862 45288.100 5.9.121.21 10.0.86.163 3787994088.090904267 3787994089.005591680 3787994089.005628622 3787994088.100062891 0 4 4 2 6 -25 0.008514 0.023209 131.188.3.220 58862 45300.040 10.0.86.163 213.172.105.106 2620639457.965499451 2620639457.860956196 4097034465.518361213 0.000000000 3 4 3 0 6 -23 0.000000 1509949440.000000 .STEP. 58862 45300.045 213.172.105.106 10.0.86.163 3787994100.040324421 3787994100.974746887 3787994100.974803586 3787994100.044992071 0 4 4 2 6 -20 0.001648 0.000809 213.172.96.14 58862 45301.036 10.0.86.163 51.75.67.47 2603862241.343136122 2603862241.407911066 4113811681.265170278 0.000000000 3 4 3 0 6 -23 0.000000 1526726656.000000 .STEP. 58862 45301.046 51.75.67.47 10.0.86.163 3787994101.035947934 3787994101.977476639 3787994101.977538857 3787994101.045554730 0 4 4 3 6 -25 0.009369 0.002960 129.70.132.36 58862 45302.064 10.0.86.163 213.172.105.106 4097034465.327065287 4097034465.697654965 4130588897.175893966 0.000000000 3 4 3 0 6 -23 0.000000 1543503872.000000 .STEP. 58862 45302.064 10.0.86.163 85.214.255.221 2587085025.097306840 2587085025.802653975 4130588897.243551496 0.000000000 3 4 3 0 6 -23 0.000000 1543503872.000000 .STEP. 58862 45302.072 213.172.105.106 10.0.86.163 3787994102.064011048 3787994103.002219000 3787994103.002296173 3787994102.071534146 0 4 4 2 6 -20 0.001648 0.000809 213.172.96.14 58862 45302.083 85.214.255.221 10.0.86.163 3787994102.064031198 3787994103.008567078 3787994103.008678016 3787994102.083063499 0 4 4 2 6 -23 0.011963 0.046188 124.216.164.14 58862 45303.096 10.0.86.163 51.75.67.47 4113811681.558536528 4113811681.548731389 4147366113.119134789 0.000000000 3 4 3 0 6 -23 0.000000 1560281088.000000 .STEP. 58862 45303.108 51.75.67.47 10.0.86.163 3787994103.096168465 3787994104.041492098 3787994104.041567690 3787994103.107718816 0 4 4 3 6 -25 0.009369 0.002960 129.70.132.36 58862 45304.096 10.0.86.163 85.214.255.221 4147366113.955935956 4130588897.264255707 4164143329.456872707 0.000000000 3 4 3 0 6 -23 0.000000 1577058304.000000 .STEP. 58862 45304.115 85.214.255.221 10.0.86.163 3787994104.096145001 3787994105.044396789 3787994105.044502776 3787994104.115178756 0 4 4 2 6 -23 0.011963 0.046219 124.216.164.14 58862 45314.072 10.0.86.163 138.201.64.208 2587085025.738188020 2587085025.905258304 37013729.149878089 0.000000000 3 4 3 0 6 -23 0.000000 1744830464.000000 .STEP. 58862 45314.081 138.201.64.208 10.0.86.163 3787994114.071920285 3787994115.035003144 3787994115.035035224 3787994114.081076858 0 4 4 2 6 -24 0.008438 0.031143 131.188.3.222 58862 45316.039 10.0.86.163 138.201.64.208 53790945.519805433 37013729.954891269 70568161.138580444 0.000000000 3 4 3 0 6 -23 0.000000 1778384896.000000 .STEP. 58862 45316.048 138.201.64.208 10.0.86.163 3787994116.039100297 3787994117.005769807 3787994117.005820531 3787994116.048254631 0 4 4 2 6 -24 0.008438 0.031174 131.188.3.222 58862 45319.058 10.0.86.163 78.46.253.198 3409168609.405931707 3409168609.357987707 120899809.477460626 0.000000000 3 4 3 0 6 -23 0.000000 1828716544.000000 .STEP. 58862 45319.067 78.46.253.198 10.0.86.163 3787994119.058047919 3787994120.030293321 3787994120.030319871 3787994119.067352633 0 4 4 2 6 -21 0.011612 0.038544 237.17.204.95 58862 45320.054 10.0.86.163 159.69.150.81 3425945825.747040982 3425945825.229344968 137677025.874599579 0.000000000 3 4 3 3 6 -23 1493499904.000000 3136356608.000000 78.46.253.198 58862 45320.062 159.69.150.81 10.0.86.163 3787994120.053755156 3787994121.026963968 3787994121.027123505 3787994120.061710944 0 4 4 2 6 -24 0.007172 0.028091 131.188.3.220 58862 45335.051 10.0.86.163 178.63.247.119 3677604065.895129859 3677604065.932350522 389335265.835038427 0.000000000 3 4 3 3 6 -23 1493499904.000000 3388014848.000000 78.46.253.198 58862 45335.061 178.63.247.119 10.0.86.163 3787994135.051098217 3787994136.053191144 3787994136.053201539 3787994135.060564965 0 4 4 2 6 -25 0.005371 0.027817 17.253.54.123 58862 45336.054 10.0.86.163 90.187.7.5 3694381281.523389100 3694381281.316005058 406112481.822644296 0.000000000 3 4 3 3 6 -23 1493499904.000000 3404792064.000000 78.46.253.198 58862 45336.077 90.187.7.5 10.0.86.163 3787994136.053979446 3787994137.060615431 3787994137.060630847 3787994136.077218155 0 4 4 1 6 -24 0.000000 0.002502 .DCFa. 58862 45344.054 10.0.86.163 213.209.109.44 3828599009.043319998 3828599009.948717419 540330209.896908346 0.000000000 3 4 3 3 6 -23 1493499904.000000 3539009792.000000 78.46.253.198 58862 45344.054 10.0.86.163 82.64.45.50 3828599009.677390991 3828599009.630694096 540330209.003369096 0.000000000 3 4 3 3 6 -23 1493499904.000000 3539009792.000000 78.46.253.198 58862 45344.066 213.209.109.44 10.0.86.163 3787994144.053857559 3787994145.070982009 3787994145.071021710 3787994144.066056797 0 4 4 2 6 -24 0.000351 0.006836 10.129.9.96 58862 45344.086 82.64.45.50 10.0.86.163 3787994144.053907156 3787994145.084690120 3787994145.084694947 3787994144.086118647 0 4 4 1 6 -25 0.000000 0.000000 .GPS. 58862 45345.042 10.0.86.163 136.243.177.133 3862153441.273351372 3845376225.388106591 557107425.584468605 0.000000000 3 4 3 3 6 -23 1493499904.000000 3539009792.000000 78.46.253.198 58862 45345.051 136.243.177.133 10.0.86.163 3787994145.041940664 3787994146.062204633 3787994146.062280472 3787994145.051393929 0 4 4 2 6 -23 0.008759 0.028427 131.188.3.222 58862 45348.082 10.0.86.163 5.9.121.21 3912485089.608893395 3895707873.733285731 607439073.385771279 0.000000000 3 4 3 3 6 -23 1493499904.000000 3589341440.000000 78.46.253.198 58862 45348.091 5.9.121.21 10.0.86.163 3787994148.081676566 3787994149.107454605 3787994149.107533267 3787994148.090983552 0 4 4 2 6 -25 0.008514 0.024124 131.188.3.220 58862 45362.057 10.0.86.163 213.172.105.106 4147366113.384698272 4130588897.816639904 842320097.147255960 0.000000000 3 4 3 3 6 -23 1493499904.000000 3824222464.000000 78.46.253.198 58862 45362.061 213.172.105.106 10.0.86.163 3787994162.056910643 3787994163.106706988 3787994163.106734188 3787994162.061432005 0 4 4 2 6 -20 0.001831 0.000244 213.172.96.14 58862 45363.056 10.0.86.163 51.75.67.47 4164143329.121805432 4147366113.845544046 859097313.130925003 0.000000000 3 4 3 3 6 -23 1493499904.000000 3840999680.000000 78.46.253.198 58862 45363.066 51.75.67.47 10.0.86.163 3787994163.055870302 3787994164.112263317 3787994164.112327436 3787994163.065631574 0 4 4 3 6 -25 0.009369 0.002991 129.70.132.36 58862 45364.096 10.0.86.163 85.214.255.221 4180920545.681768658 4164143329.872474439 875874529.252496546 0.000000000 3 4 3 3 6 -23 1493499904.000000 3857776896.000000 78.46.253.198 58862 45364.116 85.214.255.221 10.0.86.163 3787994164.096155360 3787994165.155928792 3787994165.156026089 3787994164.115799563 0 4 4 2 6 -23 0.011963 0.047119 124.216.164.14 58862 45376.079 10.0.86.163 138.201.64.208 87345377.306464970 70568161.364904049 1077201121.168582921 0.000000000 3 4 3 3 6 -23 1493499904.000000 4059103488.000000 78.46.253.198 58862 45376.089 138.201.64.208 10.0.86.163 3787994176.079103957 3787994177.157180019 3787994177.157290125 3787994176.088556430 0 4 4 2 6 -24 0.008438 0.032074 131.188.3.222 58862 45381.096 10.0.86.163 78.46.253.198 137677025.019710721 120899809.671954993 1161087201.594095057 0.000000000 3 4 3 3 6 -23 1493499904.000000 4142989568.000000 78.46.253.198 58862 45381.108 78.46.253.198 10.0.86.163 3787994181.096146977 3787994182.185581401 3787994182.185823299 3787994181.108031710 0 4 4 2 6 -21 0.011612 0.039474 237.17.204.95 58862 45382.052 10.0.86.163 159.69.150.81 154454241.904555382 137677025.833157304 1177864417.635473433 0.000000000 3 4 3 3 6 -23 1493499904.000000 4159766784.000000 78.46.253.198 58862 45382.060 159.69.150.81 10.0.86.163 3787994182.052302279 3787994183.140304250 3787994183.140388829 3787994182.060109532 0 4 4 2 6 -24 0.007172 0.029022 131.188.3.220 (In reply to Vinícius Zavam from comment #137) This is excessive and should have been in an attachment. (In reply to Vinícius Zavam from comment #136) Rather than post full binary images of your memsticks, svn rev number and svn diff from the root of the source tree used to build the images is preferred. (In reply to Cy Schubert from comment #139) yeah, sorry for the big paste before. my bad ;\ regarding the diffs and revs we are all good; we have that here already :) as I used 'generic' and all default settings to create the USB images, it's pretty easy to reproduce. Given that you tried and had the issue under 13-CURRENT on the Dell R440 and that I run 13-CURRENT on four machines without the issue here, and that you don't have the issue on non-Dell R440, Dell R440 is required to reproduce this problem. Correct? (In reply to Cy Schubert from comment #141) you are correct; maybe other machines using same board and CPU can also have this issue but I do not same such a similar hardware to test it. dmesg and more information are also available here on this PR. should you guys need any extra detail, just need to ask. tyvm! The discussion at https://lists.freebsd.org/pipermail/freebsd-current/2020-March/075472.html describes this problem. See kib@'s summary of the problem at the end of the thread : https://lists.freebsd.org/pipermail/freebsd-current/2020-March/075489.html. |
*** this issue was first noticed on FreeBSD stable/12@r354367; amd64. after updating the machine to stable/12@r354893 it still persists. bare metal, no VM. OSVERSION and KERNVERSION are both 1201502. *** it was tested in 2 different machines (same hardware with same BIOS version, drivers and so on; all up to date. dell poweredge r440). % grep -i ntp /etc/rc.conf ntpdate_enable="YES" ntpd_enable="YES" ntpd_flags="${ntpd_flags} -4 --logfile=/var/log/ntpd.log --statsdir=/etc/ntp/stats -g -g -g -G" % grep -v ^# /etc/ntp.conf restrict default limited kod nomodify notrap noquery nopeer restrict 127.0.0.1 mask 255.255.255.255 restrict 10.64.35.0 mask 255.255.255.248 nomodify notrap server ntp1 iburst prefer server ntp2 iburst prefer server ntp3 iburst server ntp4 iburst tos orphan 3 statistics clockstats cryptostats loopstats peerstats rawstats sysstats % uptime 12:39PM up 19:28, 4 users, load averages: 0.33, 0.26, 0.21 % date ; ntpq -4 -n -c peers localhost Thu Nov 21 12:39:57 CET 2019 remote refid st t when poll reach delay offset jitter ============================================================================== 10.0.5.3 134.130.4.17 2 u 14 64 1 0.172 1295.19 145.506 10.0.5.4 192.87.106.2 2 u 12 64 1 0.199 1151.60 163.782 10.0.32.248 134.130.4.17 2 u 15 64 1 1.039 1093.62 252.553 10.0.32.249 192.87.106.2 2 u 14 64 1 1.064 1084.35 263.641 % tail -n5 /var/log/ntpd.log 21 Nov 11:54:19 ntpd[62796]: kernel reports TIME_ERROR: 0x2041: Clock Unsynchronized 21 Nov 11:54:19 ntpd[62796]: kernel reports TIME_ERROR: 0x2041: Clock Unsynchronized 21 Nov 11:54:27 ntpd[62796]: Doing intital time step 21 Nov 12:07:40 ntpd[62796]: receive: Unexpected origin timestamp 0xe180ee7b.1b07155c does not match aorg 0000000000.00000000 from server@10.0.32.249 xmt 0xe180ee7c.0e6de70f 21 Nov 12:12:48 ntpd[62796]: receive: Unexpected origin timestamp 0xe180efb0.0f7b03f3 does not match aorg 0000000000.00000000 from server@10.0.32.249 xmt 0xe180efb0.c5f625d7 *** the server reported above by ntpd.log is NOT always the same; also used servers from the public pool offered by ntp.org, and others. *** stratum 1 and 2 were used to try synchonizing the clock. the machine ran with a slightly nodified 'ntp.conf' with +4 servers for a couple minutes. same behavior. *** a reboot was conducted just in case. # tail /etc/ntp/stats/*.20191121 ==> /etc/ntp/stats/loopstats.20191121 <== 58808 39269.319 0.000000000 41.447 0.000000238 0.000000 6 58808 39276.266 0.016292991 41.447 0.005760442 0.000000 6 58808 39280.272 0.024378028 41.447 0.006099659 0.000000 6 58808 39283.292 0.028349141 41.447 0.005875910 0.000000 6 58808 39284.325 0.031962872 41.447 0.005642953 0.000000 6 58808 39648.103 0.000000000 41.447 0.000000238 0.000000 6 58808 40060.057 0.000000000 41.447 0.000000238 0.000000 6 58808 40368.773 0.000000000 41.447 0.000000238 0.000003 6 58808 41071.263 0.000000000 41.447 0.000000238 0.000003 6 58808 41831.078 0.000000000 41.447 0.000000238 0.000003 6 ==> /etc/ntp/stats/peerstats.20191121 <== 58808 41832.116 10.0.5.4 9014 1.084662017 0.000208704 7.937500151 0.000000238 58808 41832.116 10.0.5.3 9014 1.084808389 0.000198845 7.937500151 0.000000238 58808 41832.117 10.0.32.249 9074 1.084358544 0.001063677 7.937500157 0.000000238 58808 41836.090 10.0.32.248 9064 1.093624959 0.001039041 7.937500157 0.000000238 58808 41861.061 10.0.5.4 9014 1.151605000 0.000199260 3.937608976 0.066942983 58808 41921.087 10.0.5.3 9014 1.290578800 0.000184648 3.937833976 0.205770411 58808 41923.097 10.0.5.3 9014 1.295191429 0.000171885 1.937678388 0.148799025 58808 42044.123 10.0.32.248 9064 1.574754317 0.000975819 0.438625303 0.321708459 58808 42045.071 10.0.5.3 961a 1.577294399 0.000142329 0.438419041 0.325921391 58808 42107.065 10.0.32.249 9374 1.720286304 0.000934281 0.188952496 0.417919861 ==> /etc/ntp/stats/rawstats.20191121 <== 58808 42045.071 10.0.5.3 10.0.5.7 3783325245.070321137 3783325246.647686701 3783325246.647714587 3783325245.070491352 0 4 4 2 6 -24 0.002029 0.038559 134.130.4.17 58808 42045.071 10.0.32.249 10.0.248.26 3783325245.070242874 3783325246.647650840 3783325246.647691908 3783325245.071362245 0 4 4 2 6 -24 0.006226 0.043961 192.87.106.2 58808 42047.107 10.0.5.7 10.0.5.4 49709281.717738837 32932065.437737468 1073119457.879780001 0.000000000 3 4 3 3 6 -22 2382364672.000000 1449198080.000000 10.0.5.3 58808 42047.107 10.0.5.4 10.0.5.7 3783325247.106494539 3783325248.688494213 3783325248.688520585 3783325247.106853646 0 4 4 2 6 -24 0.005630 0.042023 192.87.106.2 58808 42106.109 10.0.248.26 10.0.32.248 1039565025.590554219 1022787809.035393484 2062975201.462277836 0.000000000 3 4 3 3 6 -22 2382364672.000000 2422276608.000000 10.0.5.3 58808 42106.110 10.0.32.248 10.0.248.26 3783325306.108754603 3783325307.827359874 3783325307.827429942 3783325306.109918638 0 4 4 2 6 -24 0.002396 0.036011 134.130.4.17 58808 42107.064 10.0.248.26 10.0.32.249 1056342241.047421434 1039565025.799976591 2079752417.367572907 0.000000000 3 4 3 3 6 -22 2382364672.000000 2439053824.000000 10.0.5.3 58808 42107.064 10.0.5.7 10.0.5.3 1056342241.541501084 1039565025.647339527 2079752417.948276881 0.000000000 3 4 3 3 6 -22 2382364672.000000 2439053824.000000 10.0.5.3 58808 42107.064 10.0.5.3 10.0.5.7 3783325307.063579735 3783325308.784316616 3783325308.784336750 3783325307.063787127 0 4 4 2 6 -24 0.002029 0.039490 134.130.4.17 58808 42107.065 10.0.32.249 10.0.248.26 3783325307.063508592 3783325308.784262036 3783325308.784329157 3783325307.064509994 0 4 4 2 6 -24 0.006226 0.044891 192.87.106.2 ==> /etc/ntp/stats/sysstats.20191121 <== 58808 1051.535 3600 288 236 288 0 0 0 0 0 0 0 58808 4660.020 3600 287 231 287 0 0 0 0 0 0 0 58808 8266.163 3600 281 227 281 0 0 0 0 0 0 0 58808 11875.331 3600 284 231 284 0 0 0 0 0 0 0 58808 15484.618 3600 288 235 288 0 0 0 0 0 0 0 58808 19091.952 3600 284 227 284 0 0 0 0 0 0 0 58808 22701.535 3600 284 238 284 0 0 0 0 0 0 0 58808 26308.724 3600 282 228 282 0 0 0 0 0 0 0 58808 29917.975 3600 286 233 286 0 0 0 0 0 0 0 58808 33526.016 3600 272 227 272 0 0 0 0 0 0 0 % date ; ntpq -4 -n -c peers localhost Thu Nov 21 12:53:22 CET 2019 remote refid st t when poll reach delay offset jitter ============================================================================== *10.0.5.3 134.130.4.17 2 u 11 64 1 0.168 973.772 232.305 +10.0.5.4 192.87.106.2 2 u 9 64 1 0.195 978.173 278.795 +10.0.32.248 134.130.4.17 2 u 12 64 1 1.071 684.348 162.969 +10.0.32.249 192.87.106.2 2 u 11 64 1 1.061 624.044 207.852 *** other machines running either older revisions of stable/12 or releng/12.0 and releng/12.1 are not affected.