Bug 242137 - Dell R440 loses time on FreeBSD-12-STABLE following r352517, exceeding the NTP threshold of 500 PPM to correct the error.
Summary: Dell R440 loses time on FreeBSD-12-STABLE following r352517, exceeding the NT...
Status: Closed Overcome By Events
Alias: None
Product: Base System
Classification: Unclassified
Component: bin (show other bugs)
Version: 12.0-STABLE
Hardware: amd64 Any
: --- Affects Some People
Assignee: Cy Schubert
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-11-21 11:54 UTC by Vinícius Zavam
Modified: 2020-03-13 20:26 UTC (History)
3 users (show)

See Also:


Attachments
13-0-amd64-r354935_dmesg.txt (92.12 KB, text/plain)
2019-11-25 09:32 UTC, Vinícius Zavam
no flags Details
13-0-amd64-r354935_pciconf.txt (25.78 KB, text/plain)
2019-11-25 09:32 UTC, Vinícius Zavam
no flags Details
STABLE_12_r354893_ntpd.log (5.43 KB, text/plain)
2019-11-26 11:58 UTC, Vinícius Zavam
no flags Details
STABLE_12_r354367_dmesg.boot.txt (58.24 KB, text/plain)
2019-12-04 10:38 UTC, Vinícius Zavam
no flags Details
STABLE_12_r354367_ntpq_rootdisp.png (72.16 KB, image/png)
2019-12-04 13:29 UTC, Vinícius Zavam
no flags Details
ntpdate_offsetvars.txt (152.92 KB, text/plain)
2020-01-07 09:47 UTC, Vinícius Zavam
no flags Details
svnlite_diff_stable_12__r352513_r352517.patch (2.59 KB, patch)
2020-01-13 10:37 UTC, Vinícius Zavam
no flags Details | Diff
svnlite_diff_stable_12__r352513_r352519.patch (8.93 KB, patch)
2020-01-13 10:38 UTC, Vinícius Zavam
no flags Details | Diff
svnlite_diff_stable_12__r352517_r352519.patch (6.97 KB, patch)
2020-01-13 10:38 UTC, Vinícius Zavam
no flags Details | Diff
DIFF_base_stable_12_r356668__without__r352865.patch (2.74 KB, patch)
2020-01-13 15:50 UTC, Vinícius Zavam
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Vinícius Zavam freebsd_committer 2019-11-21 11:54:46 UTC
*** this issue was first noticed on FreeBSD stable/12@r354367; amd64. after updating the machine to stable/12@r354893 it still persists. bare metal, no VM. OSVERSION and KERNVERSION are both 1201502.

*** it was tested in 2 different machines (same hardware with same BIOS version, drivers and so on; all up to date. dell poweredge r440).

% grep -i ntp /etc/rc.conf
ntpdate_enable="YES"
ntpd_enable="YES"
ntpd_flags="${ntpd_flags} -4 --logfile=/var/log/ntpd.log --statsdir=/etc/ntp/stats -g -g -g -G"

% grep -v ^# /etc/ntp.conf 
restrict default limited kod nomodify notrap noquery nopeer
restrict 127.0.0.1 mask 255.255.255.255
restrict 10.64.35.0 mask 255.255.255.248 nomodify notrap
server ntp1 iburst prefer
server ntp2 iburst prefer
server ntp3 iburst
server ntp4 iburst
tos orphan 3
statistics clockstats cryptostats loopstats peerstats rawstats sysstats

% uptime
12:39PM  up 19:28, 4 users, load averages: 0.33, 0.26, 0.21

% date ; ntpq -4 -n -c peers localhost
Thu Nov 21 12:39:57 CET 2019
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 10.0.5.3        134.130.4.17     2 u   14   64    1    0.172  1295.19 145.506
 10.0.5.4        192.87.106.2     2 u   12   64    1    0.199  1151.60 163.782
 10.0.32.248     134.130.4.17     2 u   15   64    1    1.039  1093.62 252.553
 10.0.32.249     192.87.106.2     2 u   14   64    1    1.064  1084.35 263.641

% tail -n5 /var/log/ntpd.log
21 Nov 11:54:19 ntpd[62796]: kernel reports TIME_ERROR: 0x2041: Clock Unsynchronized
21 Nov 11:54:19 ntpd[62796]: kernel reports TIME_ERROR: 0x2041: Clock Unsynchronized
21 Nov 11:54:27 ntpd[62796]: Doing intital time step
21 Nov 12:07:40 ntpd[62796]: receive: Unexpected origin timestamp 0xe180ee7b.1b07155c does not match aorg 0000000000.00000000 from server@10.0.32.249 xmt 0xe180ee7c.0e6de70f
21 Nov 12:12:48 ntpd[62796]: receive: Unexpected origin timestamp 0xe180efb0.0f7b03f3 does not match aorg 0000000000.00000000 from server@10.0.32.249 xmt 0xe180efb0.c5f625d7

*** the server reported above by ntpd.log is NOT always the same; also used servers from the public pool offered by ntp.org, and others.

*** stratum 1 and 2 were used to try synchonizing the clock. the machine ran with a slightly nodified 'ntp.conf' with +4 servers for a couple minutes. same behavior.

*** a reboot was conducted just in case. 

# tail /etc/ntp/stats/*.20191121
==> /etc/ntp/stats/loopstats.20191121 <==
58808 39269.319 0.000000000 41.447 0.000000238 0.000000 6
58808 39276.266 0.016292991 41.447 0.005760442 0.000000 6
58808 39280.272 0.024378028 41.447 0.006099659 0.000000 6
58808 39283.292 0.028349141 41.447 0.005875910 0.000000 6
58808 39284.325 0.031962872 41.447 0.005642953 0.000000 6
58808 39648.103 0.000000000 41.447 0.000000238 0.000000 6
58808 40060.057 0.000000000 41.447 0.000000238 0.000000 6
58808 40368.773 0.000000000 41.447 0.000000238 0.000003 6
58808 41071.263 0.000000000 41.447 0.000000238 0.000003 6
58808 41831.078 0.000000000 41.447 0.000000238 0.000003 6

==> /etc/ntp/stats/peerstats.20191121 <==
58808 41832.116 10.0.5.4 9014 1.084662017 0.000208704 7.937500151 0.000000238
58808 41832.116 10.0.5.3 9014 1.084808389 0.000198845 7.937500151 0.000000238
58808 41832.117 10.0.32.249 9074 1.084358544 0.001063677 7.937500157 0.000000238
58808 41836.090 10.0.32.248 9064 1.093624959 0.001039041 7.937500157 0.000000238
58808 41861.061 10.0.5.4 9014 1.151605000 0.000199260 3.937608976 0.066942983
58808 41921.087 10.0.5.3 9014 1.290578800 0.000184648 3.937833976 0.205770411
58808 41923.097 10.0.5.3 9014 1.295191429 0.000171885 1.937678388 0.148799025
58808 42044.123 10.0.32.248 9064 1.574754317 0.000975819 0.438625303 0.321708459
58808 42045.071 10.0.5.3 961a 1.577294399 0.000142329 0.438419041 0.325921391
58808 42107.065 10.0.32.249 9374 1.720286304 0.000934281 0.188952496 0.417919861

==> /etc/ntp/stats/rawstats.20191121 <==
58808 42045.071 10.0.5.3 10.0.5.7 3783325245.070321137 3783325246.647686701 3783325246.647714587 3783325245.070491352 0 4 4 2 6 -24 0.002029 0.038559 134.130.4.17
58808 42045.071 10.0.32.249 10.0.248.26 3783325245.070242874 3783325246.647650840 3783325246.647691908 3783325245.071362245 0 4 4 2 6 -24 0.006226 0.043961 192.87.106.2
58808 42047.107 10.0.5.7 10.0.5.4 49709281.717738837 32932065.437737468 1073119457.879780001 0.000000000 3 4 3 3 6 -22 2382364672.000000 1449198080.000000 10.0.5.3
58808 42047.107 10.0.5.4 10.0.5.7 3783325247.106494539 3783325248.688494213 3783325248.688520585 3783325247.106853646 0 4 4 2 6 -24 0.005630 0.042023 192.87.106.2
58808 42106.109 10.0.248.26 10.0.32.248 1039565025.590554219 1022787809.035393484 2062975201.462277836 0.000000000 3 4 3 3 6 -22 2382364672.000000 2422276608.000000 10.0.5.3
58808 42106.110 10.0.32.248 10.0.248.26 3783325306.108754603 3783325307.827359874 3783325307.827429942 3783325306.109918638 0 4 4 2 6 -24 0.002396 0.036011 134.130.4.17
58808 42107.064 10.0.248.26 10.0.32.249 1056342241.047421434 1039565025.799976591 2079752417.367572907 0.000000000 3 4 3 3 6 -22 2382364672.000000 2439053824.000000 10.0.5.3
58808 42107.064 10.0.5.7 10.0.5.3 1056342241.541501084 1039565025.647339527 2079752417.948276881 0.000000000 3 4 3 3 6 -22 2382364672.000000 2439053824.000000 10.0.5.3
58808 42107.064 10.0.5.3 10.0.5.7 3783325307.063579735 3783325308.784316616 3783325308.784336750 3783325307.063787127 0 4 4 2 6 -24 0.002029 0.039490 134.130.4.17
58808 42107.065 10.0.32.249 10.0.248.26 3783325307.063508592 3783325308.784262036 3783325308.784329157 3783325307.064509994 0 4 4 2 6 -24 0.006226 0.044891 192.87.106.2

==> /etc/ntp/stats/sysstats.20191121 <==
58808 1051.535 3600 288 236 288 0 0 0 0 0 0 0
58808 4660.020 3600 287 231 287 0 0 0 0 0 0 0
58808 8266.163 3600 281 227 281 0 0 0 0 0 0 0
58808 11875.331 3600 284 231 284 0 0 0 0 0 0 0
58808 15484.618 3600 288 235 288 0 0 0 0 0 0 0
58808 19091.952 3600 284 227 284 0 0 0 0 0 0 0
58808 22701.535 3600 284 238 284 0 0 0 0 0 0 0
58808 26308.724 3600 282 228 282 0 0 0 0 0 0 0
58808 29917.975 3600 286 233 286 0 0 0 0 0 0 0
58808 33526.016 3600 272 227 272 0 0 0 0 0 0 0

% date ; ntpq -4 -n -c peers localhost
Thu Nov 21 12:53:22 CET 2019
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*10.0.5.3        134.130.4.17     2 u   11   64    1    0.168  973.772 232.305
+10.0.5.4        192.87.106.2     2 u    9   64    1    0.195  978.173 278.795
+10.0.32.248     134.130.4.17     2 u   12   64    1    1.071  684.348 162.969
+10.0.32.249     192.87.106.2     2 u   11   64    1    1.061  624.044 207.852

*** other machines running either older revisions of stable/12 or releng/12.0 and releng/12.1 are not affected.
Comment 1 Vinícius Zavam freebsd_committer 2019-11-21 11:57:28 UTC
% ntpq -4 -n -c peers ntp1
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
-81.173.193.222  130.149.17.21    2 u  560 1024  377    1.318    1.931   0.427
-81.173.193.223  193.67.79.202    2 u  501 1024  377    1.325    1.580   0.148
*134.130.4.17    .GPS.            1 u  349 1024  377    2.087   -0.340   0.078
+134.130.5.17    .GPS.            1 u  679 1024  377    2.122   -0.343   0.090
+129.143.2.23    .GPS.            1 u  930 1024  377    7.557   -0.407   0.119
-129.143.2.33    .PZF.            1 u  446 1024  377    7.556   -0.612   0.055
-194.109.6.2     193.67.79.202    2 u  544 1024  377    5.820   -0.565   0.369
-213.136.0.252   .PPS.            1 u  659 1024  377    6.473   -0.132   0.958
-192.53.103.103  .PTB.            1 u  383 1024  377   13.873    1.432   1.090
-10.0.5.4        192.87.106.2     2 s  764 1024  376    0.218   -0.158   0.105

% ntpq -4 -n -c peers ntp2
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
-81.173.193.222  130.149.17.21    2 u  752 1024  377    1.197    2.366   0.228
-81.173.193.223  193.67.79.202    2 u  802 1024  377    1.457    1.595   0.211
+134.130.4.17    .GPS.            1 u  605 1024  377    2.156   -0.173   0.048
+134.130.5.17    .GPS.            1 u  966 1024  377    2.186   -0.194   0.032
+129.143.2.23    .GPS.            1 u  118 1024  377    7.666   -0.239   0.080
-129.143.2.33    .PZF.            1 u  503 1024  377    7.487   -0.448   0.056
-194.109.6.2     193.67.79.202    2 u  393 1024  377    5.515   -0.296   0.063
*192.87.106.2    .GPS.            1 u  601 1024  377    5.719   -0.084   0.091
+192.87.36.4     .MRS.            1 u  128 1024  377    8.280    0.013   0.084
-10.0.5.3        134.130.4.17     2 s  245 1024  377    0.202    0.150   0.092
Comment 2 Vinícius Zavam freebsd_committer 2019-11-21 14:16:23 UTC
*** same hardware running releng/12.1 works fine as well
Comment 3 Vinícius Zavam freebsd_committer 2019-11-21 16:34:40 UTC
% ntpd --version
ntpd 4.2.8p12-a (1)
Comment 4 Vinícius Zavam freebsd_committer 2019-11-22 09:57:00 UTC
*** 12.1-STABLE amd64 r354893 1201502
Comment 5 Vinícius Zavam freebsd_committer 2019-11-22 10:38:48 UTC
% uptime
11:38AM  up 1 day, 18:23, 5 users, load averages: 0.53, 0.34, 0.27

% cat /var/db/ntp/ntpd.drift
48.229
Comment 6 Vinícius Zavam freebsd_committer 2019-11-22 11:32:51 UTC
recently booted the machine using 'FreeBSD-13.0-CURRENT-amd64-20191121-r354935-memstick.img' and it behaves the same as stable/12@r354893.

https://svnweb.freebsd.org/base/head/contrib/ntp/ntpd/?view=log suggests that 'ntpd' should be version 4.2.8p13, it's still 4.2.8p12 (1). just like under stable/12 (12.1-STABLE).

how to reproduce?
  1. boot usb stick with FreeBSD-13.0-CURRENT-amd64-20191121-r354935;
  2. enter shell mode;
  3. get your network settings done;
  4. run `service ntpd onestart` and wait.
Comment 7 Vinícius Zavam freebsd_committer 2019-11-22 11:33:37 UTC
FreeBSD freebsd-HEAD-13-0 13.0-CURRENT FreeBSD 13.0-CURRENT #0 r354935: Thu Nov 21 08:28:22 UTC 2019     root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC  amd64

11:33AM  up 1 hr, 2 users, load averages: 0.14, 0.17, 0.21

ntpq> lpeers
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 0.freebsd.pool. .POOL.          16 p    -   64    0    0.000    0.000   0.000
+193.1.12.167    230.0.204.150    2 u   59   64    1   26.073  492.557 266.495
+82-64-45-50.sub .GPS.            1 u   58   64    1   32.215  493.964 265.248
+216.6.2.70 (up2 195.219.14.21    2 u   56   64    1   91.754  490.716 265.393
+server.samoylyk 161.143.24.141   2 u   52   64    1   24.803  500.755 299.806
+v35005.php-frie 131.188.3.223    2 u   40   64    1    4.615  528.354 314.264
+stratum2-3.NTP. 129.70.130.70    2 u   20   64    7   11.671  566.456 419.030
+time02.fal.code 192.53.103.108   2 u   18   64    7    9.590  570.200 421.709
+162.159.200.123 10.48.8.4        3 u   16   64    7    1.900  573.828 420.288
+213.251.53.187  195.66.241.10    2 u   16   64    7    5.425  573.531 422.477
+ntp1.rrze.uni-e .DCFp.           1 u   16   64    7    7.907  572.864 426.169
+x8d1ee404.agdsn 85.220.190.246   3 u   16   64    7   19.868  571.959 421.706
*ntp2.rrze.uni-e .MBGh.           1 u   19   64   37    7.964  567.335 384.663

ntpq> sysinfo
associd=0 status=0613 leap_none, sync_ntp, 1 event, spike_detect,
system peer:        ntp2.rrze.uni-erlangen.de:123
system peer mode:   client
leap indicator:     00
stratum:            2
log2 precision:     -23
root delay:         7.964
root dispersion:    617.125
reference ID:       131.188.3.222
reference time:     e18245a6.3f00f24a  Fri, Nov 22 2019 11:31:50.246
system jitter:      42.259079
clock jitter:       33.033
clock wander:       0.000
broadcast delay:    -50.000
symm. auth. delay:  0.000

ntpq> version
ntpq 4.2.8p12-a (1)
Comment 8 Vinícius Zavam freebsd_committer 2019-11-22 16:58:32 UTC
that's odd: running the same 13.0-CURRENT r354935 live stick in another machine (regular PC, with an Intel Core i3-3110M@2.4Ghz) it works! the CPU on the server is an Intel Xeon Bronze 3204@1.9Ghz

current status: till 12.1-RELEASE the box with the Xeon works fine, running stable/12 (r354367 and r354893) or HEAD (r354935) NTP does not work.
Comment 9 Vinícius Zavam freebsd_committer 2019-11-25 09:32:12 UTC
Created attachment 209404 [details]
13-0-amd64-r354935_dmesg.txt

13-0-amd64-r354935_dmesg.txt
Comment 10 Vinícius Zavam freebsd_committer 2019-11-25 09:32:29 UTC
Created attachment 209405 [details]
13-0-amd64-r354935_pciconf.txt

13-0-amd64-r354935_pciconf.txt
Comment 11 Vinícius Zavam freebsd_committer 2019-11-25 09:35:23 UTC
root@13-0-amd64-r354935:~ # date
Mon Nov 25 09:33:27 UTC 2019

root@13-0-amd64-r354935:~ # uptime
 9:33AM  up 18 mins, 2 users, load averages: 0.05, 0.14, 0.10

ntpq> version
ntpq 4.2.8p12-a (1)

ntpq> sysinfo
associd=0 status=0613 leap_none, sync_ntp, 1 event, spike_detect,
system peer:        zarkon.mcl.gg:123
system peer mode:   client
leap indicator:     00
stratum:            3
log2 precision:     -23
root delay:         42.766
root dispersion:    756.559
reference ID:       94.16.113.67
reference time:     e1861e40.639d7150  Mon, Nov 25 2019  9:32:48.389
system jitter:      110.350937
clock jitter:       26.918
clock wander:       0.000
broadcast delay:    -50.000
symm. auth. delay:  0.000

ntpq> lpeers
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 0.freebsd.pool. .POOL.          16 p    -   64    0    0.000    0.000   0.000
+frank.askja.de  124.216.164.14   2 u  115   64    1    2.378  549.955 329.626
+spacys.de       130.149.17.8     2 u  114   64    1   26.613  542.582 321.933
+185.90.160.100  100.10.69.89     2 u  113   64    1    5.412  553.136 328.887
+meinekiste.de   170.231.250.158  2 u  115   64    1    9.228  550.012 330.670
+245.ip-51-75-74 82.161.139.11    2 u   26   64   17   17.817  713.543 514.645
+server.spnr.de  124.216.164.14   2 u   47   64   73   25.461  667.287 492.443
*zarkon.mcl.gg   131.188.3.222    2 u   53   64   77   35.762  650.561 439.307
+hotel.zq1.de    161.62.157.173   3 u   46   64   77   30.966  666.341 453.826
Comment 12 Vinícius Zavam freebsd_committer 2019-11-25 15:05:47 UTC
using pkg's ntp package didn't workaround the issue; ntp-4.2.8p13_5
Comment 13 Vinícius Zavam freebsd_committer 2019-11-25 17:44:28 UTC
openntpd-6.2p3_4,2

% ntpctl -s all
4/4 peers valid, constraint offset 0s, clock unsynced, clock offset is 536.865ms

peer
   wt tl st  next  poll          offset       delay      jitter
62.116.130.3 from pool pool.ntp.org
    1 10  2   23s   30s       419.828ms    24.961ms    17.435ms
131.188.3.221 from pool pool.ntp.org
    1 10  1   11s   33s       386.589ms    25.516ms    17.142ms
188.68.36.203 from pool pool.ntp.org
    1 10  2   20s   30s       406.697ms    24.534ms    20.615ms
185.244.195.159 from pool pool.ntp.org
    1 10  2   24s   33s       410.479ms    23.648ms    17.262ms
Comment 14 Vinícius Zavam freebsd_committer 2019-11-25 17:49:48 UTC
chrony-3.5

chronyc> tracking
Reference ID    : 8AC95ABD (ns1.your-ns.de)
Stratum         : 3
Ref time (UTC)  : Mon Nov 25 17:48:09 2019
System time     : 0.976312995 seconds slow of NTP time
Last offset     : -0.117800929 seconds
RMS offset      : 0.117547080 seconds
Frequency       : 0.000 ppm slow
Residual freq   : +198.180 ppm
Skew            : 1000000.000 ppm
Root delay      : 0.064190254 seconds
Root dispersion : 64.806434631 seconds
Update interval : 64.7 seconds
Leap status     : Normal
Comment 15 Cy Schubert freebsd_committer 2019-11-25 21:08:13 UTC
Read this carefully. There are a number of outputs being requested here.

Your network or upstream has issues. Compare offset and jitter of the server that has the error with the server that does not. You've only been able to reach once (or 7 times). Compere this with those that work.

Can we have output of cat /var/db/ntp/ntp.drift please? Large drift in conjunction with the inability to contact each ntp sever is the cause of this error.

You should see synchronization flags (*, #, +, -) in your ntpq output. We see none. You're not synchronizing. Again, this points to probable problems with either your local network, upstream, or denied by your firewall.

----

Try adding rlimit memlock 32 to ntp.conf. (This will not work with ASLR enabled.) This was the only change to ntp prior to r354367. (With ASLR enabled, mlock() caused a segfault and stack corruption.)

How much RAM is installed on this machaine? How much swap is configured? How much swap is in use?

----

Please answer all of the questions above. Again:

1. Can you ping your ntp servers? What is the round trip time?
2. Is UDP/123 open?
3. What is in ntp.drift?
4. How much RAM is in the machine?
5. Is swap being used and is ntpd using some of that swap?
6. Does setting memlock in ntp.conf do anything for you?
Comment 16 Vinícius Zavam freebsd_committer 2019-11-25 22:12:57 UTC
(In reply to Cy Schubert from comment #15)

hey, thanks for sharing some thoughts here cy@. appreciated!

I'm curious to check and understand how you meant that it's a network issue; as other machines running in the same network sync their clock waaaaay faster and with no issues like this one reported here. would really like to dig more into it.

the ntp.drift was posted on "Comment 5": its value was 48.229 after 1d18h of uptime with ntpd running.

"Comment 1" has the output of two of the ntp servers that I use to synchonize this ntpd instance. AFAIK, both are running fine - ntp1 is on the same network as the problematic machine.

at the end of the initial report you can see the flags, just like on Comments 7 and 11 - they are from the same machine (once running stable/12, later running head).

so, as I do have a very identical box to use it was moved to a different location and I cloned the system from the original server and had the same issue in another location (free of firewalling though); when I boot 12.1-RELEASE it just works as expected.

-----

1. yes; min/avg/max/mdev = 0.119/0.216/0.284/0.057 ms
2. yes;
3. 48.229 (posted before);
4. 32G;
5. no;
6. did not change a thing, used default (FreeBSD's) ntp.conf during tests only.
Comment 17 Cy Schubert freebsd_committer 2019-11-26 03:47:51 UTC
I assume you restarted ntpd after setting rlimit memlock 32 ?
Comment 18 Cy Schubert freebsd_committer 2019-11-26 04:08:44 UTC
1. Stop ntpd.

2. tcpdump -i YOUR_EXTERNAL_INTERFACE udp port 123

    or if you prefer not to post here

2. tcpdump -i YOUR_EXTERNAL_INTERFACE -w ntp.pcap udp port 123

   and email the output file to my freebsd.org email address (I'll review using wireshark)

3. start ntpd

4. Then either post the output or email the file to me. I suspect ntpd is sending but not receiving.
Comment 19 Vinícius Zavam freebsd_committer 2019-11-26 09:47:54 UTC
(In reply to Cy Schubert from comment #17)

I did not test it. I used only default ntp.conf shipped with FreeBSD without any changes. but yes, if I change the ntp.conf I do restart its daemon right after :)

forgot to mention one thing though: what I did try was changing 'kern.timecounter.hardware' to other values instead os just TSC.

I am working here to get a side-by-side scenario with 2 machines before starting the tcpdump - still curious why: the machine works fine when I run releng/12.1 as I mentioned before

thanks!
Comment 20 Vinícius Zavam freebsd_committer 2019-11-26 10:58:26 UTC
I'm running 'tcpdump' and playing with 2 different machines, both running HEAD/amd64 r354935. one is the R440 (bad guy) and the other one is just a regular PC.

the other R440 I left running with stable/12 r354893 for a couple of days has these information to share:



% uptime
11:50AM  up 5 days, 18:22, 1 user, load averages: 0.27, 0.24, 0.24

% cat /var/db/ntp/ntpd.drift
80.938

% ntpq -4 -n -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 10.0.5.3        134.130.4.17     2 u   59   64    1    0.338  311.293   0.000
 10.0.5.4        192.87.106.2     2 u    8   64    1    0.182  420.868  78.854
 10.0.32.248     134.130.4.17     2 u   60   64    1    1.105  307.941   0.000
 10.0.32.249     192.87.106.2     2 u   10   64    1    0.980  416.475  77.317

here we do not see the flags, but if I try to get the peers list from ***other servers*** (also running FreeBSD) in the very same network and using the very same ntp.conf, we have the following:



12.1-RELEASE-p1% ntpq -4 -n -p 
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*10.0.5.3        134.130.4.17     2 u  121 1024  377    0.650    0.747   0.460
+10.0.5.4        192.87.106.2     2 u  627 1024  377    0.560   -0.909   0.834
-10.0.32.248     134.130.4.17     2 u   37 1024  377    2.279   -0.666   3.731
+10.0.32.249     192.87.106.2     2 u  487 1024  377    1.602   -0.848   0.955



12.1-STABLE-r352727% ntpq -4 -n -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*10.0.5.3        134.130.4.17     2 u   99  128  377    0.081    0.315   0.232
+10.0.5.4        192.87.106.2     2 u  111  128  377    0.150   -1.349   0.132
+10.0.32.248     134.130.4.17     2 u    5  128  377    0.889   -0.554   7.056
-10.0.32.249     192.87.106.2     2 u   12  128  377    0.888   -1.363   1.518
Comment 21 Vinícius Zavam freebsd_committer 2019-11-26 11:58:27 UTC
Created attachment 209450 [details]
STABLE_12_r354893_ntpd.log

attaching the ntpd.log from yesterday, which is running for about 5 days.
Comment 22 Vinícius Zavam freebsd_committer 2019-11-26 12:11:09 UTC
booted an USB memstick and let it run for a while, using all the defaults we have shipped with the live/install image:

root@HEADamd64r354935-a:~ # uptime
12:08PM  up  3:18, 2 users, load averages: 0.17, 0.25, 0.24

root@HEADamd64r354935-a:~ # grep \$FreeBSD /etc/ntp.conf
# $FreeBSD: head/usr.sbin/ntp/ntpd/ntp.conf 352304 2019-09-13 20:20:05Z cy $

root@HEADamd64r354935-a:~ # grep -v -E '^#|^$' /etc/ntp.conf 
tos minclock 3 maxclock 6
pool 0.freebsd.pool.ntp.org iburst
restrict default limited kod nomodify notrap noquery nopeer
restrict source  limited kod nomodify notrap noquery
restrict 127.0.0.1
restrict ::1
leapfile "/var/db/ntpd.leap-seconds.list"

root@HEADamd64r354935-a:~ # cat /var/db/ntp/ntpd.drift
500.000

root@HEADamd64r354935-a:~ # ntpq -4 -n -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 0.freebsd.pool. .POOL.          16 p    -   64    0    0.000    0.000   0.000
+116.203.209.35  237.17.204.95    2 u   39   64    3    7.661  844.631 295.042
+80.67.16.8      131.188.3.221    2 u   38   64    3    1.761  1420.93 511.244
+144.76.70.233   152.103.15.66    2 u   37   64    3    9.098  1189.64 313.973
+193.30.120.245  192.53.103.108   2 u   36   64    3    7.786  847.167 298.156
+203.107.6.88    10.137.38.86     2 u   22   64    3  364.706  1125.61 289.341
+195.46.37.22    194.146.251.101  2 u    7   64    3   33.843  904.179 301.022
*213.136.0.252   .PPS.            1 u    9   64    3    6.729  1011.97 254.233
+163.237.218.19  .GPS.            1 u    5   64    3   98.093  1249.28 299.505
+188.125.64.6    212.82.106.33    2 u    3   64    3   23.957  1252.51 329.107
+85.195.227.162  124.216.164.14   2 u   66   64    3    9.857  795.025 292.792
+185.17.70.106   162.23.41.55     2 u    5   64    3   14.402  1136.51 264.795
+193.140.100.40  160.45.10.8      2 u    4   64    3   62.183  1136.77 266.951

root@HEADamd64r354935-a:~ # nc -nzvu 85.195.227.162 123
Connection to 85.195.227.162 123 port [udp/*] succeeded!

root@HEADamd64r354935-a:~ # nc -nzvu 163.237.218.19 123
Connection to 163.237.218.19 123 port [udp/*] succeeded!

root@HEADamd64r354935-a:~ # nc -nzvu 131.188.3.221 123
Connection to 131.188.3.221 123 port [udp/*] succeeded!
Comment 23 Vinícius Zavam freebsd_committer 2019-11-26 12:14:54 UTC
now the other machine that works (same version and rev from machine A, and same ntp.conf):

root@HEADamd64r354935-b:~ # uptime 
12:12PM  up  3:27, 2 users, load averages: 0.12, 0.14, 0.09

root@HEADamd64r354935-b:~ # cat /var/db/ntp/ntpd.drift
11.226

root@HEADamd64r354935-b:~ # ntpq -4 -n -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 0.freebsd.pool. .POOL.          16 p    -   64    0    0.000    0.000   0.000
+145.239.0.227   131.188.3.223    2 u   46   64  377   17.364  -14.710   1.288
+217.144.138.234 131.188.3.220    2 u   23   64  377   11.204  -14.875   2.140
-88.99.216.194   131.188.3.223    2 u    6   64  377   16.726  -14.850   5.395
-176.9.1.211     161.62.157.173   3 u  100   64  376   15.619  -16.407   7.332
*62.128.1.18     213.95.151.123   2 u   24   64  377   11.255  -15.058   6.604

root@HEADamd64r354935-b:~ # nc -nzvu 88.99.216.194 123
Connection to 88.99.216.194 123 port [udp/*] succeeded!

root@HEADamd64r354935-b:~ # nc -nzvu 217.144.138.234 123
Connection to 217.144.138.234 123 port [udp/*] succeeded!
Comment 24 Vinícius Zavam freebsd_committer 2019-11-26 12:17:34 UTC
netcat from my workstation to both machines (first machine A, than machine B):

% nc -nzvu 10.0.86.163 123
Connection to 10.0.86.163 123 port [udp/*] succeeded!

% nc -nzvu 10.0.128.203 123
Connection to 10.0.128.203 123 port [udp/*] succeeded!

PS: mailed you the tcpdump from both machines
Comment 25 Cy Schubert freebsd_committer 2019-11-26 13:10:05 UTC
Looking at tcpdump outputs before I go to $JOB, I'll put this here so you can check this out:

10.0.128.203 and 10.0.86.163 are serving up garbage. The NTP packets say they are stratum 0 time servers (unspecified or invalid) and their reference, origin, and receive timestamps are Feb 7, 2036 (the fields are 0's).

What are the A and B servers IP's again?
Comment 26 Vinícius Zavam freebsd_committer 2019-11-26 13:39:30 UTC
(In reply to Cy Schubert from comment #25)

10.0.86.163 == a
10.0.128.203 == b
Comment 27 Vinícius Zavam freebsd_committer 2019-11-26 13:44:38 UTC
both, A and B, are using 'pool 0.freebsd.pool.ntp.org' and are not meant to really work as NTP servers, just use ntp as client - like the default ntp.conf we have right now (Comment 22).

the actual NTP servers I have are all working just fine though.
Comment 28 Cy Schubert freebsd_committer 2019-11-26 13:54:49 UTC
That doesn't explain why you are being served up bad time by the servers in your 10.0 network. That's your problem. Fix that and your problem is fixed.

0.freebsd.pool.ntp.org does not contain 10.0 servers in its definition. YOur ntp.conf from reply #1 does not include 0.freebsd.pool.ntp.org:

restrict default limited kod nomodify notrap noquery nopeer
restrict 127.0.0.1 mask 255.255.255.255
restrict 10.64.35.0 mask 255.255.255.248 nomodify notrap
server ntp1 iburst prefer
server ntp2 iburst prefer
server ntp3 iburst
server ntp4 iburst
tos orphan 3
statistics clockstats cryptostats loopstats peerstats rawstats sysstats
Comment 29 Vinícius Zavam freebsd_committer 2019-11-26 14:08:31 UTC
in total, I have 3 machines: 2 are R440 (which are impacted with this issues to synchronize time) and 1 is just an offset box.

machine X is 10.0.5.7 (running since 5 days, with NTP sync issues);
machine A is 10.0.86.163 (very same hw as machine X, live img running);
machine B is 10.0.128.203 (offset box, live img running).

ntp servers 1, 2, 3, and 4 are fine: 2 of them are on 10.0.5.x, and 2 are on 10.0.248.x. no issues at all. these are the current internal stratum 2 servers.

machines X and A can't work as expected and use ntpd to synchronize their clocks; as we see here the drifts are kinda high, offset and jitter are pretty bad although the delay/latency to reach internal and external NTP servers is not a problem.

hope that gives all a better picture of the scenario here. maybe I should have written it before.
Comment 30 Vinícius Zavam freebsd_committer 2019-11-26 14:13:01 UTC
(In reply to Cy Schubert from comment #28)

no no no. this config you pasted is definitely *NOT* used to sync against pool.ntp.org!

the actual ntp.conf for internal use is:

-----
restrict        default limited kod nomodify notrap noquery nopeer
restrict        127.0.0.1 mask 255.255.255.255
restrict        10.0.0.0  mask 255.255.0.0     nomodify notrap
restrict        10.64.0.0 mask 255.255.0.0     nomodify notrap
server  ntp1       iburst prefer
server  ntp2       iburst prefer
server  ntp3       iburst # prefer
server  ntp4       iburst # prefer
tos     orphan  3
-----

the 'statistics' was added just to collect data; the restrict was modified before but THIS ONE is the one used in prod for this environment
Comment 31 Vinícius Zavam freebsd_committer 2019-11-26 14:37:09 UTC
current test setup now on machine A and B: both bare metal with live images running HEAD amd64 r354935.

# ntp.conf
tos minclock 3 maxclock 6
pool 0.freebsd.pool.ntp.org iburst
restrict default limited kod nomodify notrap noquery nopeer
restrict source  limited kod nomodify notrap noquery
restrict 127.0.0.1
restrict ::1
leapfile "/var/db/ntpd.leap-seconds.list"

after getting network to work, just ran

# service ntpd onestart
Comment 32 Vinícius Zavam freebsd_committer 2019-11-26 14:49:07 UTC
short update on the 'ntpq -p' for machine A and B (with their flags):
----------

root@HEADamd64r354935-a:~ # uptime
 2:45PM  up  5:55, 2 users, load averages: 0.17, 0.28, 0.27

root@HEADamd64r354935-a:~ # ntpq -4 -n -p 
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 0.freebsd.pool. .POOL.          16 p    -   64    0    0.000    0.000   0.000
#82.100.248.10   194.97.129.17    2 u   54   64    1    4.631  528.815 222.540
+131.234.137.63  .DCF.            1 u   50   64    1   13.021  881.310 243.876
#138.201.16.225  130.149.17.21    2 u   50   64    1    9.245  532.426 226.107
#5.189.146.13    131.188.3.220    2 u   50   64    1   14.836  655.488 141.082
#176.9.166.35    192.53.103.108   2 u   34   64    1    9.306  685.282 147.836
#193.30.120.245  192.53.103.108   2 u   16   64    1    7.779  943.407 268.613
+173.249.58.145  131.188.3.222    2 u   12   64    1   15.856  951.403 270.882
+37.120.184.82   131.188.3.222    2 u   56   64    1    7.635  869.279 239.843
+188.68.36.203   131.188.3.221    2 u   57   64    1    7.623  752.591 142.400
+80.67.16.8      3.121.254.221    3 u   58   64    1    1.762  754.617 142.360
+213.239.239.166 237.17.204.95    2 u   57   64    1    9.279  756.224 146.244
*5.199.135.170   130.149.17.21    2 u   56   64    1    2.139  868.705 239.148

root@HEADamd64r354935-a:~ # cat /var/db/ntp/ntpd.drift 
500.000
----------

root@HEADamd64r354935-b:~ # uptime
 2:46PM  up  6:01, 2 users, load averages: 0.10, 0.14, 0.09

root@HEADamd64r354935-b:~ # ntpq -4 -n -p 
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 0.freebsd.pool. .POOL.          16 p    -   64    0    0.000    0.000   0.000
+178.162.199.162 131.188.3.223    2 u   15  128  377   20.890   -9.873  26.352
+144.76.0.164    193.67.79.202    2 u    1  128  377   16.949   -9.525   2.543
*90.187.99.165   .PPS.            1 u  123  128  377   32.826  -12.698   2.729
+217.91.44.17    192.168.100.15   2 u  117  128  377   23.217  -10.839   3.000
+80.153.195.191  192.168.100.9    2 u  124  128  377   25.459   -7.415   2.383

root@HEADamd64r354935-b:~ # cat /var/db/ntp/ntpd.drift 
6.134
Comment 33 Vinícius Zavam freebsd_committer 2019-11-26 14:54:46 UTC
regarding the comments about "network issues": how would that explain a pretty normal behavior of NTP running 12.1-RELEASE on the machine A (on the very same network)?
Comment 34 Cy Schubert freebsd_committer 2019-11-26 20:14:29 UTC
After briefly looking at the pcap files you sent me, it's not a network issue but the ntp servers on your 10.0 network are serving bad time. They say they are stratum 0 time servers, which is invalid and the three time fields in the ntp packets are zero.

The ntpq -p from your A and B servers look good.
Comment 35 Vinícius Zavam freebsd_committer 2019-11-27 09:52:36 UTC
(In reply to Cy Schubert from comment #34)

% host ntp
ntp.localdomain has address 10.0.5.4
ntp.localdomain.com has address 10.0.32.249
ntp.localdomain.com has address 10.0.32.248
ntp.localdomain.com has address 10.0.5.3

% ntpdate -q ntp
server 10.0.5.4, stratum 2, offset -0.000039, delay 0.02678
server 10.0.32.249, stratum 2, offset 0.000023, delay 0.02589
server 10.0.32.248, stratum 2, offset -0.000080, delay 0.02583
server 10.0.5.3, stratum 2, offset -0.000569, delay 0.02702
27 Nov 09:43:16 ntpdate[11451]: adjust time server 10.0.32.248 offset -0.000080 sec
-----

% ntpq -4 -n -p ntp1
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
-81.173.193.222  131.188.3.220    2 u  979 1024  377    1.343    2.425   0.145
-81.173.193.223  193.67.79.202    2 u  736 1024  377    1.176    1.530   0.307
*134.130.4.17    .GPS.            1 u  890 1024  377    2.168    0.380   0.305
+134.130.5.17    .GPS.            1 u  808 1024  377    2.055    0.293   0.369
+129.143.2.23    .GPS.            1 u  352 1024  377    7.571    0.311   0.302
+129.143.2.33    .PZF.            1 u  811 1024  377    7.481    0.196   0.192
-194.109.6.2     193.79.237.14    2 u  293 1024  377    5.773    0.022   0.277
+213.136.0.252   .PPS.            1 u  962 1024  377    6.243    0.202   0.511
+192.53.103.103  .PTB.            1 u  927 1024  377   10.518    0.404   1.809
-10.0.5.4        192.87.106.2     2 s  349 1024  337    0.183    0.729   0.162

% ntpq -4 -n -p ntp2
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
-81.173.193.222  130.149.17.21    2 u  416 1024  377    1.172    1.517   0.120
-81.173.193.223  193.67.79.202    2 u  726 1024  377    1.031    0.762   0.294
+134.130.4.17    .GPS.            1 u  255 1024  377    2.217   -0.387   0.176
+134.130.5.17    .GPS.            1 u  718 1024  377    2.150   -0.310   0.104
-129.143.2.23    .GPS.            1 u  149 1024  377    7.572   -0.480   0.180
-129.143.2.33    .PZF.            1 u  852 1024  377    7.477   -0.650   0.218
-194.109.6.2     193.67.79.202    2 u  568 1024  377    5.456   -0.565   0.046
*192.87.106.2    .GPS.            1 u  542 1024  377    5.897   -0.054   0.100
+192.87.36.4     .MRS.            1 u   36 1024  377    8.288   -0.209   0.210
-10.0.5.3        134.130.4.17     2 s  367 1024  176    0.066   -0.670   0.205

% ntpq -4 -n -p ntp3
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
-81.173.193.222  131.188.3.220    2 u  961 1024  377    1.851    1.497   0.454
-81.173.193.223  193.67.79.202    2 u  102 1024  377    1.499    0.702   2.060
*134.130.4.17    .GPS.            1 u  489 1024  377    2.481   -0.365   4.585
+134.130.5.17    .GPS.            1 u  912 1024  377    2.419   -0.361   0.426
+129.143.2.23    .GPS.            1 u  371 1024  377    7.941   -0.306   0.312
+129.143.2.33    .PZF.            1 u  994 1024  377    7.829   -0.547   0.330
-194.109.6.2     193.67.79.202    2 u  462 1024  377    6.535   -0.867   6.943
-213.136.0.252   .PPS.            1 u  394 1024  377    8.033   -0.973   5.084
+192.53.103.103  .PTB.            1 u  346 1024  377   10.727   -0.306   1.693
-10.0.32.249     192.87.106.2     2 s  580 1024  376    0.180    0.157   0.329

% ntpq -4 -n -p ntp4
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
-81.173.193.222  131.188.3.220    2 u  781 1024  377    2.001    1.389   6.224
-81.173.193.223  193.67.79.202    2 u  277 1024  377    1.477    0.594   1.523
+134.130.4.17    .GPS.            1 u  511 1024  377    2.537   -0.467   8.575
+134.130.5.17    .GPS.            1 u   41 1024  377    2.467   -0.515   1.370
-129.143.2.23    .GPS.            1 u  577 1024  377    7.978   -0.511   0.063
-129.143.2.33    .PZF.            1 u  376 1024  377    7.928   -0.759   2.418
-194.109.6.2     193.67.79.202    2 u  812 1024  377    6.086   -0.628   0.657
*192.87.106.2    .GPS.            1 u    3 1024  377    6.066   -0.334   0.128
+192.87.36.4     .MRS.            1 u  125 1024  377    8.419   -0.356   0.106
-10.0.32.248     134.130.4.17     2 s  316 1024  377    0.141   -0.138   0.321
-----

% ntpq -4 -n -c sysinfo ntp1
associd=0 status=0615 leap_none, sync_ntp, 1 event, clock_sync,
system peer:        134.130.4.17:123
system peer mode:   client
leap indicator:     00
stratum:            2
log2 precision:     -24
root delay:         2.050
root dispersion:    19.509
reference ID:       134.130.4.17
reference time:     e188c440.20a64f14  Wed, Nov 27 2019  9:45:36.127
system jitter:      0.495904
clock jitter:       0.264
clock wander:       0.055
broadcast delay:    -50.000
symm. auth. delay:  0.010

% ntpq -4 -n -c sysinfo ntp2
associd=0 status=0615 leap_none, sync_ntp, 1 event, clock_sync,
system peer:        192.87.106.2:123
system peer mode:   client
leap indicator:     00
stratum:            2
log2 precision:     -24
root delay:         5.897
root dispersion:    45.148
reference ID:       192.87.106.2
reference time:     e188bd76.faadd7b2  Wed, Nov 27 2019  9:16:38.979
system jitter:      0.100174
clock jitter:       0.138
clock wander:       0.013
broadcast delay:    -50.000
symm. auth. delay:  0.008

% ntpq -4 -n -c sysinfo ntp3
associd=0 status=0615 leap_none, sync_ntp, 1 event, clock_sync,
system peer:        134.130.4.17:123
system peer mode:   client
leap indicator:     00
stratum:            2
log2 precision:     -24
root delay:         2.481
root dispersion:    34.979
reference ID:       134.130.4.17
reference time:     e188c1b9.3abded05  Wed, Nov 27 2019  9:34:49.229
system jitter:      4.585486
clock jitter:       0.170
clock wander:       0.037
broadcast delay:    -50.000
symm. auth. delay:  0.000

% ntpq -4 -n -c sysinfo ntp4
associd=0 status=0615 leap_none, sync_ntp, 1 event, clock_sync,
system peer:        192.87.106.2:123
system peer mode:   client
leap indicator:     00
stratum:            2
log2 precision:     -24
root delay:         6.066
root dispersion:    21.975
reference ID:       192.87.106.2
reference time:     e188c3a1.1ddeb161  Wed, Nov 27 2019  9:42:57.116
system jitter:      0.128426
clock jitter:       0.153
clock wander:       0.014
broadcast delay:    -50.000
symm. auth. delay:  0.000
Comment 36 Vinícius Zavam freebsd_committer 2019-11-27 09:56:45 UTC
(In reply to Vinícius Zavam from comment #35)

DISCLAIMER: previous 'host' command returns

% host ntp
ntp.localdomain has address 10.0.5.4
ntp.localdomain has address 10.0.32.249
ntp.localdomain has address 10.0.32.248
ntp.localdomain has address 10.0.5.3

just replaced domain's name; let's focus on the NTP thing. DNS is definitely working!
Comment 37 Vinícius Zavam freebsd_committer 2019-11-27 10:01:06 UTC
I mailed you the pcap file which I ran in my workstation, using these 4 servers.
Comment 38 Vinícius Zavam freebsd_committer 2019-11-27 11:46:20 UTC
*** ran the "Platform Specific Bootable ISO, PER440,19.10.00" to update BIOS and other firmwares, but no updates were applied - the box is already with latest stuff. https://www.dell.com/support/home/us/en/04/drivers/driversdetails?driverid=1098x&oscode=naa&productcode=poweredge-r440 

this is machine X and A ^
Comment 39 Cy Schubert freebsd_committer 2019-11-27 20:48:18 UTC
Need a reset here:

Can you list the IP addresses of your 12.1, A and B servers, please. 

Also list ntp.conf for each.
Comment 40 Vinícius Zavam freebsd_committer 2019-11-28 14:06:12 UTC
machine X; 10.0.5.7 (stable/12@r354893, 12.1-STABLE)
machine A; 10.0.86.163 (head@r354935)
machine B; 10.0.128.203 (head@r354935)

# grep -v -E '^#|^$' /etc/ntp.conf 
tos minclock 3 maxclock 6
pool 0.freebsd.pool.ntp.org iburst
restrict default limited kod nomodify notrap noquery nopeer
restrict source  limited kod nomodify notrap noquery
restrict 127.0.0.1
restrict ::1
leapfile "/var/db/ntpd.leap-seconds.list"
Comment 41 Vinícius Zavam freebsd_committer 2019-11-29 15:11:32 UTC
(In reply to Cy Schubert from comment #25)

regarding the timestamp with 'Feb 7 2036' you mentioned before: I've tcpdumped other FreeBSD machines and they all have the same "issue".

these recently tested FreeBSD machines are not even in the same location/country, and therefore can't be influenced by the other machines/servers mentioned before.

they run the following revisions: r353654 e r354990 and r354992 (HEAD/amd64). they're all using the same default ntp.conf from FreeBSD, just like the one pasted on Comment 40.

this is how the time stamping looks like:

Network Time Protocol (NTP Version 4, client)
    Flags: 0xe3, Leap Indicator: unknown (clock unsynchronized), Version number: NTP Version 4, Mode: client
        11.. .... = Leap Indicator: unknown (clock unsynchronized) (3)
        ..10 0... = Version number: NTP Version 4 (4)
        .... .011 = Mode: client (3)
    [Response In: 2]
    Peer Clock Stratum: unspecified or invalid (0)
    Peer Polling Interval: 6 (64 seconds)
    Peer Clock Precision: 0.000000 seconds
    Root Delay: 0.000000 seconds
    Root Dispersion: 0.000000 seconds
    Reference ID: (Initialization)
    Reference Timestamp: Feb  7, 2036 06:28:16.000000000 UTC
    Origin Timestamp: Feb  7, 2036 06:28:16.000000000 UTC
    Receive Timestamp: Feb  7, 2036 06:28:16.000000000 UTC
    Transmit Timestamp: Nov 29, 2019 14:45:11.501112827 UTC
-----

Network Time Protocol (NTP Version 4, client)
    Flags: 0x23, Leap Indicator: no warning, Version number: NTP Version 4, Mode: client
        00.. .... = Leap Indicator: no warning (0)
        ..10 0... = Version number: NTP Version 4 (4)
        .... .011 = Mode: client (3)
    [Response In: 58]
    Peer Clock Stratum: secondary reference (2)
    Peer Polling Interval: 6 (64 seconds)
    Peer Clock Precision: 0.000000 seconds
    Root Delay: 0.009277 seconds
    Root Dispersion: 0.940109 seconds
    Reference ID: 76.127.35.142
    Reference Timestamp: Nov 29, 2019 14:45:18.493365262 UTC
    Origin Timestamp: Feb  7, 2036 06:28:16.000000000 UTC
    Receive Timestamp: Feb  7, 2036 06:28:16.000000000 UTC
    Transmit Timestamp: Nov 29, 2019 14:45:20.484546625 UTC
Comment 42 Vinícius Zavam freebsd_committer 2019-12-04 10:38:00 UTC
Created attachment 209680 [details]
STABLE_12_r354367_dmesg.boot.txt
Comment 44 Vinícius Zavam freebsd_committer 2019-12-04 11:55:36 UTC
the 'rlimit memlock 32' option was not added to the ntp.conf on my side because it seems to be the default already.

from the ntp.conf(5) manpage:

             memlock Nmegabytes
                     Specify the number of megabytes of memory that should be
                     allocated and locked.  Probably only available under
                     Linux, this option may be useful when dropping root (the
                     -i option).  The default is 32 megabytes on non-Linux
                     machines, and -1 under Linux.  -1 means "do not lock the
                     process into memory".  0 means "lock whatever memory the
                     process wants into memory".
Comment 45 Vinícius Zavam freebsd_committer 2019-12-04 13:29:08 UTC
Created attachment 209689 [details]
STABLE_12_r354367_ntpq_rootdisp.png

12.1-STABLE, stable/12 [r354367] is the dark blue line.

used `ntpq -4 -n -c rl` to collect data from *all* these 5 machines.
Comment 46 Cy Schubert freebsd_committer 2019-12-04 13:41:21 UTC
(In reply to Vinícius Zavam from comment #44)
rlimit memlock 32 is not the default. It was removed last summer because it caused ntp to segfault with ASLR and stack gap != 0. There was a discussion about this on -current at the time.
Comment 47 Cy Schubert freebsd_committer 2019-12-04 13:42:39 UTC
Added Harlan Stenn of ntp.org to this discussion.
Comment 48 Vinícius Zavam freebsd_committer 2019-12-04 13:55:57 UTC
(In reply to Cy Schubert from comment #46)

oh, so... we could update the manpage; I might take a time to patch it later if we need to. thanks for reporting it - I missed that thread. 

I am going to add it to the .conf and restart the daemon. thanks!
Comment 49 Vinícius Zavam freebsd_committer 2019-12-04 14:21:02 UTC
same.

% ntpdate -4 -q -d 10.0.5.7
 4 Dec 14:19:20 ntpdate[14809]: ntpdate 4.2.8p12-a (1)
transmit(10.0.5.7)
receive(10.0.5.7)
10.0.5.7: Server dropped: strata too high

server 10.0.5.7, port 123
stratum 16, precision -22, leap 11, trust 000
refid [STEP], root delay 0.000000, root dispersion 0.000519
reference time:      (no time)
originate timestamp: e1923ee7.37e78eb9  Wed, Dec  4 2019 14:19:19.218
transmit timestamp:  e1923ee8.24faa500  Wed, Dec  4 2019 14:19:20.144
delay 0.02658, dispersion 0.00000, offset -0.926634

 4 Dec 14:19:20 ntpdate[14809]: no server suitable for synchronization found

% ntpq -4 -n -c rl 10.0.5.7
associd=0 status=c018 leap_alarm, sync_unspec, 1 event, no_sys_peer,
version="ntpd 4.2.8p12-a (1)", processor="amd64",
system="FreeBSD/12.1-STABLE", leap=11, stratum=16, precision=-22,
rootdelay=0.000, rootdisp=0.780, refid=STEP, reftime=(no time),
clock=e1923ef9.0dc12b58  Wed, Dec  4 2019 14:19:37.053, peer=0, tc=6,
mintc=3, offset=0.000000, frequency=128.357, sys_jitter=0.000238,
clk_jitter=0.000, clk_wander=0.000, tai=37, leapsec=201701010000,
expire=202006280000
Comment 50 Vinícius Zavam freebsd_committer 2019-12-04 14:52:25 UTC
booted the FreeBSD-12.1-STABLE-amd64-20191127-r355118-mini-memstick.img and performed `service ntpd onestart` (after getting a work network connection, of course) and was presented to the very same behavior :"(

https://download.freebsd.org/ftp/snapshots/amd64/amd64/ISO-IMAGES/12.1/FreeBSD-12.1-STABLE-amd64-20191127-r355118-mini-memstick.img
Comment 51 Vinícius Zavam freebsd_committer 2019-12-06 10:09:28 UTC
I am adding a few outputs here from truss and ktrace, which I ran in a FreeBSD HEAD@r355121 (amd64 bare metal hardware; dell poweredge r440, as reported before)

https://share.riseup.net/#lzApqrgFltk9J4rlH11Zdg
>>> freebsd-head-r355121-amd64_ktrace_output_SERVICE_ntpd_ONESTART.bin

https://share.riseup.net/#LVlKzFFoHiylOMkfu58xng
>>> ktrace_output_ntpq.bin

https://share.riseup.net/#SQCa-pCEZmTi2wB4nX8O1Q
>>> freebsd-head-r355121-amd64_truss_output_SERVICE_ntpd_ONESTART.txt

https://share.riseup.net/#Dk9Kn_Wipb-aXNkN6l-BYA
>>> truss_output_ntpq.txt
Comment 52 Vinícius Zavam freebsd_committer 2019-12-06 10:46:51 UTC
% ntpq -4 -n -c rl -wp 127.0.0.1
associd=0 status=c613 leap_alarm, sync_ntp, 1 event, spike_detect,
version="ntpd 4.2.8p12-a (1)", processor="amd64",
system="FreeBSD/13.0-CURRENT", leap=11, stratum=3, precision=-23,
rootdelay=17.795, rootdisp=1311.617, refid=213.239.239.164,
reftime=e194ad60.0e89a5e8  Fri, Dec  6 2019 10:35:12.056,
clock=e194aeb2.99e66d49  Fri, Dec  6 2019 10:40:50.601, peer=14176, tc=6,
mintc=3, offset=0.000000, frequency=500.000, sys_jitter=173.114733,
clk_jitter=0.000, clk_wander=0.000, tai=37, leapsec=201701010000,
expire=201912280000

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 0.freebsd.pool.ntp.org
                 .POOL.          16 p    -   64    0    0.000    0.000   0.000
+195.50.171.101  192.53.103.108   2 u   49   64    7    5.545  580.506 382.202
+35.158.196.249  192.53.103.108   2 u   51   64    7    4.671  576.939 382.105
#5.9.121.21      131.188.3.222    2 u   44   64    7    9.112  585.911 382.935
#78.46.241.151   213.239.239.165  3 u   49   64    7    9.312  454.438 479.780
+81.7.16.52      237.17.204.95    2 u   30   64    7   18.855  736.585 321.421
#94.16.114.254   5.9.121.21       3 u   21   64    7    7.765  869.343 305.652
#144.76.96.7     205.46.178.169   2 u   13   64    7    9.029  460.459 528.144
+193.141.27.6    192.53.103.108   2 u   14   64    7    6.719  872.974 303.831
#144.76.59.37    195.145.119.188  2 u   49   64    7    9.213  452.879 480.766
+159.69.150.81   237.17.204.95    2 u   46   64    7    7.735  809.909 292.674
+136.243.7.20    0.242.242.75     3 u   18   64    7    9.098  750.452 325.521
*213.239.239.164 205.46.178.169   2 u   16   64   17    7.723  1215.27 493.678

KTRACE: https://share.riseup.net/#0ZHiHkGHnc7rBcWTenvmeQ
TRUSS: https://share.riseup.net/#MtAe5XFYrG81DV_MCUHMOQ

root@freebsd-head-r355121-amd64:~ # grep -E -v '^$|^#' /etc/ntp.conf 
tos minclock 3 maxclock 6
pool 0.freebsd.pool.ntp.org iburst
restrict default limited kod nomodify notrap noquery nopeer
restrict source  limited kod nomodify notrap noquery
restrict 127.0.0.1
restrict ::1
leapfile "/var/db/ntpd.leap-seconds.list"
Comment 53 Vinícius Zavam freebsd_committer 2019-12-06 10:50:26 UTC
# ntpdate -4 -q -d 127.0.0.1
 6 Dec 10:45:11 ntpdate[3328]: ntpdate 4.2.8p12-a (1)
transmit(127.0.0.1)
receive(127.0.0.1)
127.0.0.1: Server dropped: leap not in sync

server 127.0.0.1, port 123
stratum 3, precision -23, leap 11, trust 000
refid [144.76.59.37], root delay 0.017899, root dispersion 2.040878
reference time:      e194afb0.dc30c0aa  Fri, Dec  6 2019 10:45:04.860
originate timestamp: e194afb7.7c01ea59  Fri, Dec  6 2019 10:45:11.484
transmit timestamp:  e194afb7.7bf25b56  Fri, Dec  6 2019 10:45:11.484
delay 0.02589, dispersion 0.00000, offset 0.000051

 6 Dec 10:45:11 ntpdate[3328]: no server suitable for synchronization found
----------

# ntpdate -4 -q -d 127.0.0.1
 6 Dec 10:45:36 ntpdate[3329]: ntpdate 4.2.8p12-a (1)
transmit(127.0.0.1)
receive(127.0.0.1)
127.0.0.1: Server dropped: leap not in sync

server 127.0.0.1, port 123
stratum 3, precision -23, leap 11, trust 000
refid [5.9.121.21], root delay 0.017548, root dispersion 2.071915
reference time:      e194afb4.d40dab7a  Fri, Dec  6 2019 10:45:08.828
originate timestamp: e194afd0.d54ad348  Fri, Dec  6 2019 10:45:36.833
transmit timestamp:  e194afd0.d5431d47  Fri, Dec  6 2019 10:45:36.833
delay 0.02573, dispersion 0.00000, offset -0.000005

 6 Dec 10:45:36 ntpdate[3329]: no server suitable for synchronization found
----------

https://share.riseup.net/#B_i8WV9SfSUSEJpRCun5rw
>>> ktrace -d -i -f /var/my_ntpd/ktrace_output_ntpq.txt 

https://share.riseup.net/#Ox3sAwTrZUjdCWrbu11djQ
>>> truss -o /var/my_ntpd/truss_output_ntpq.txt -a -f -d -D -e -H
Comment 54 Cy Schubert freebsd_committer 2019-12-06 20:48:38 UTC
(In reply to Cy Schubert from comment #46)
I've already updated the base version file and the man pages in base and the ports. There is no sense submitting patches.
Comment 55 Vinícius Zavam freebsd_committer 2019-12-09 08:26:03 UTC
(In reply to Cy Schubert from comment #54)

https://svnweb.freebsd.org/base/head/contrib/ntp/ntpd/ntp.conf.5man?view=markup#l3106

it still points out that 32mb is the default for non-Linux systems. so, upstream might be getting a patch for that or I just do not follow?
Comment 56 Cy Schubert freebsd_committer 2019-12-09 12:44:05 UTC
(In reply to Vinícius Zavam from comment #55)
See src r355388 and ports r519022.
Comment 57 Vinícius Zavam freebsd_committer 2019-12-09 13:54:39 UTC
(In reply to Cy Schubert from comment #56)

my bad though. sorry, I looked under /contrib/ntp instead of /usr/sbin/ntp. thanks!
Comment 58 Vinícius Zavam freebsd_committer 2019-12-09 13:55:55 UTC
(In reply to Vinícius Zavam from comment #57)

I meant to write 'base/usr.sbin/ntp' here (sorry for all the SPAM, just want to clarify)
Comment 59 Cy Schubert freebsd_committer 2019-12-09 20:50:13 UTC
I've asked our upstream ntp.org to look at this PR.  They have asked
about the packets in your rawstats file that contain the "Unexpected
origin timestamp 0xe180e455.d15bf3fb".  They say that the message you
see indicates that you are getting a 2nd incoming packet with the same
origin timestamp, and this message is alerting you to this situation.
Whether or not this PR is valid depends on what's going on with the root
cause of your "Unexpected origin timestamp" message.
Comment 60 Vinícius Zavam freebsd_committer 2019-12-10 10:15:23 UTC
(In reply to Cy Schubert from comment #59)

I will start the test scenario again (using the memstick live image) and collect you these stats. by the end of the day I should mail them to you. 

thanks!
Comment 61 Vinícius Zavam freebsd_committer 2019-12-10 12:54:48 UTC
(In reply to Cy Schubert from comment #59)

incoming.
Comment 62 Vinícius Zavam freebsd_committer 2019-12-16 13:46:32 UTC
hello there again, a quick update on this one:

  - someone pointed that iDRAC settings could interfere on networking stuffs on these kind of hardware - but I was not that convinced; anyway I did reset *ALL* BIOS and iDRAC settings to its defaults and got nothing new. still having issues :)
Comment 63 Cy Schubert freebsd_committer 2019-12-17 06:07:29 UTC
Rawstats doesn't look right. There is consistency to the problem.

Can you post the output of:

sysctl kern.timecounter
sysctl kern.eventtimer
sysctl kern.clockrate
sysctl hw.clockrate
sysctl kern.hz

After the system has been up for a day: date.

List all the contents of rc.conf.
Comment 64 Cy Schubert freebsd_committer 2019-12-17 06:30:44 UTC
Also,

cat /var/db/ntp/ntpd.drift
Comment 65 Vinícius Zavam freebsd_committer 2019-12-17 10:16:14 UTC
as I did restart one of the ntpd processes on a stable/12 machine yesterday, I can share its information now (after posting this, I should setup a lab with the same hardware running HEAD and get the data tomorrow).

FreeBSD 12.1-STABLE r354893 amd64 1201502

% cat /var/db/ntp/ntpd.drift
134.745

hw.clockrate: 1900
kern.clockrate: { hz = 1000, tick = 1000, profhz = 8128, stathz = 127 }
kern.eventtimer.periodic: 0
kern.eventtimer.timer: LAPIC
kern.eventtimer.idletick: 0
kern.eventtimer.singlemul: 2
kern.eventtimer.choice: LAPIC(600) HPET(550) HPET1(440) HPET2(440) i8254(100) RTC(0)
kern.eventtimer.et.HPET2.quality: 440
kern.eventtimer.et.HPET2.frequency: 24000000
kern.eventtimer.et.HPET2.flags: 3
kern.eventtimer.et.HPET1.quality: 440
kern.eventtimer.et.HPET1.frequency: 24000000
kern.eventtimer.et.HPET1.flags: 3
kern.eventtimer.et.HPET.quality: 550
kern.eventtimer.et.HPET.frequency: 24000000
kern.eventtimer.et.HPET.flags: 7
kern.eventtimer.et.i8254.quality: 100
kern.eventtimer.et.i8254.frequency: 1193182
kern.eventtimer.et.i8254.flags: 1
kern.eventtimer.et.RTC.quality: 0
kern.eventtimer.et.RTC.frequency: 32768
kern.eventtimer.et.RTC.flags: 17
kern.eventtimer.et.LAPIC.quality: 600
kern.eventtimer.et.LAPIC.frequency: 1900000000
kern.eventtimer.et.LAPIC.flags: 7
kern.hz: 1000
kern.timecounter.tsc_shift: 1
kern.timecounter.smp_tsc_adjust: 0
kern.timecounter.smp_tsc: 1
kern.timecounter.invariant_tsc: 1
kern.timecounter.fast_gettime: 1
kern.timecounter.tick: 1
kern.timecounter.choice: ACPI-fast(900) HPET(950) i8254(0) TSC(1000) dummy(-1000000)
kern.timecounter.hardware: TSC
kern.timecounter.alloweddeviation: 5
kern.timecounter.timehands_count: 2
kern.timecounter.stepwarnings: 0
kern.timecounter.tc.ACPI-fast.quality: 900
kern.timecounter.tc.ACPI-fast.frequency: 3579545
kern.timecounter.tc.ACPI-fast.counter: 1271069
kern.timecounter.tc.ACPI-fast.mask: 16777215
kern.timecounter.tc.HPET.quality: 950
kern.timecounter.tc.HPET.frequency: 24000000
kern.timecounter.tc.HPET.counter: 1219858571
kern.timecounter.tc.HPET.mask: 4294967295
kern.timecounter.tc.i8254.quality: 0
kern.timecounter.tc.i8254.frequency: 1193182
kern.timecounter.tc.i8254.counter: 49961
kern.timecounter.tc.i8254.mask: 65535
kern.timecounter.tc.TSC.quality: 1000
kern.timecounter.tc.TSC.frequency: 1900000000
kern.timecounter.tc.TSC.counter: 1044650998
kern.timecounter.tc.TSC.mask: 4294967295
Comment 66 Vinícius Zavam freebsd_committer 2019-12-17 10:26:00 UTC
(In reply to Vinícius Zavam from comment #65)

running since Dec 16 18:11 CET
Comment 67 Vinícius Zavam freebsd_committer 2019-12-17 10:47:20 UTC
should you guys need graphs of this particular machine's status (stable/12@r354893) please check https://imgur.com/a/RsHZZiw
Comment 68 Cy Schubert freebsd_committer 2019-12-17 12:50:14 UTC
Capture some data: ntpq -p and list that here.

Stop ntpd.

Clear the ntpd.drift file.

Run ntpdate.

Start ntpd.

Capture data after 30 minutes: ntpq -p

Let it run for 24 hours.

Capture data: ntpq -p
Comment 69 Vinícius Zavam freebsd_committer 2019-12-17 13:51:05 UTC
(In reply to Cy Schubert from comment #68)

# date ; ntpq -4 -n -c rl -wp localhost ; service ntpd stop ; rm /var/db/ntp/ntpd.drift ; ntpdate ntp2.prod.localdomain.de ; service ntpd start ; sleep 1800 ; date ; ntpq -4 -n -c rl -wp localhost

----------
Tue Dec 17 14:49:28 CET 2019
associd=0 status=c613 leap_alarm, sync_ntp, 1 event, spike_detect,
version="ntpd 4.2.8p12-a (1)", processor="amd64",
system="FreeBSD/12.1-STABLE", leap=11, stratum=3, precision=-23,
rootdelay=6.309, rootdisp=1096.697, refid=10.0.5.3,
reftime=e1a35a1d.493a2f96  Tue, Dec 17 2019 14:43:57.286,
clock=e1a35b68.c0de8af2  Tue, Dec 17 2019 14:49:28.753, peer=36487, tc=6,
mintc=3, offset=0.000000, frequency=134.745, sys_jitter=272.750727,
clk_jitter=0.000, clk_wander=0.000, tai=37, leapsec=201701010000,
expire=202006280000

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*10.0.5.3        213.136.0.252    2 u    9   64   17    0.266  533.253 412.658
+10.0.5.4        192.53.103.103   2 u   13   64    7    0.108  397.921 461.285
+10.0.32.248     213.136.0.252    2 u    5   64   17    1.026  1103.46 589.876
-10.0.32.249     213.136.0.252    2 u   13   64    7    1.022  806.084 394.811
Stopping ntpd.
Waiting for PIDS: 72250, 72250.
17 Dec 14:49:37 ntpdate[24764]: step time server 10.0.5.4 offset 1.271688 sec
Starting ntpd.
----------

in 30min I should return back to paste the result of the second 'ntpq -4 -n -c rl -wp localhost'
Comment 70 Cy Schubert freebsd_committer 2019-12-17 14:14:11 UTC
Wow that's a large offset and jitter.

It should look more like this:

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 0.freebsd.pool. .POOL.          16 p    -   64    0    0.000    0.000   0.001
+s206-75-147-25. 192.168.10.254   2 u    3  128  377   31.505    0.469   0.650
*ntp1.torix.ca   .PTP0.           1 u   65   64  377   63.519    0.246   1.018
+nms.switch.ca   206.108.0.131    2 u   63  128  377   28.519   -0.945   0.667
-bhs2.vxnd.net   206.108.0.133    2 u  129  128  377   82.168    7.876   1.091
-draco.spiderspa 213.251.128.249  2 u    7  256  377   83.962    5.603   0.434


or this:

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 0.freebsd.pool. .POOL.          16 p    -   64    0    0.000    0.000   0.000
*cwfw            206.108.0.131    2 u   44   64  377    0.295   -0.384   0.282
+cwsys           10.1.1.254       3 s   49   64  376    0.379   -0.699   0.197
+bob             10.1.1.254       3 s   10   64  377    0.441   -0.507   0.249
-nms.switch.ca   206.108.0.131    2 u   40   64  377   28.827   -1.526   0.339
-162.159.200.1   10.28.8.252      3 u   40   64  377   11.723    2.663   0.846

LAN or upstream network connectivity looks to be a problem.

ping -c 10 each of your upstream ntp servers (though I suspect they block ICMP ECHO) but worth testing anyway. You could try a traceroute instead.
Comment 71 Vinícius Zavam freebsd_committer 2019-12-17 14:38:29 UTC
Tue Dec 17 15:19:40 CET 2019

associd=0 status=c018 leap_alarm, sync_unspec, 1 event, no_sys_peer,
version="ntpd 4.2.8p12-a (1)", processor="amd64",
system="FreeBSD/12.1-STABLE", leap=11, stratum=16, precision=-22,
rootdelay=0.000, rootdisp=3.015, refid=STEP, reftime=(no time),
clock=e1a3627c.39550433  Tue, Dec 17 2019 15:19:40.223, peer=0, tc=6,
mintc=3, offset=0.000000, frequency=500.000, sys_jitter=0.000238,
clk_jitter=0.000, clk_wander=0.000, tai=37, leapsec=201701010000,
expire=202006280000

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 10.0.32.248     134.130.4.17     2 u   14   64    1    1.022  202.981 274.644
 10.0.5.4        213.136.0.252    2 u   13   64    1    0.113  435.020 149.384
 10.0.32.249     213.136.0.252    2 u   13   64    1    0.990  203.009 347.601

# cat /var/db/ntp/ntpd.drift
cat: /var/db/ntp/ntpd.drift: No such file or directory
Comment 72 Vinícius Zavam freebsd_committer 2019-12-17 14:40:25 UTC
(In reply to Cy Schubert from comment #70)

yes, it is indeed! if I do check the same on my other machines it looks like this:

% ntpq -4 -n -c rl -wp localhost
associd=0 status=0618 leap_none, sync_ntp, 1 event, no_sys_peer,
version="ntpd 4.2.8p12-a (1)", processor="amd64",
system="FreeBSD/12.1-RELEASE-p1", leap=00, stratum=2, precision=-24,
rootdelay=10.586, rootdisp=25.481, refid=192.53.103.108,
reftime=e1a36486.c3041de6  Tue, Dec 17 2019 15:28:22.761,
clock=e1a3671a.594f2eaf  Tue, Dec 17 2019 15:39:22.348, peer=41868,
tc=10, mintc=3, offset=-0.237943, frequency=24.158, sys_jitter=0.115137,
clk_jitter=0.149, clk_wander=0.004, tai=37, leapsec=201701010000,
expire=202006280000

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
-134.130.4.17    .MRS.            1 u  278 1024  377    2.505   -0.233   0.203
+134.130.5.17    .MRS.            1 u  237 1024  377    2.310   -0.159   0.311
+129.143.2.23    .GPS.            1 u  203 1024  377    7.585   -0.202   0.257
-129.143.2.33    .PZF.            1 u  654 1024  377    7.484   -0.574   0.460
-192.87.106.2    .GPS.            1 u  895 1024  377    6.320   -0.071   0.762
*192.53.103.108  .PTB.            1 u  660 1024  377   10.586   -0.288   1.472
+192.53.103.103  .PTB.            1 u  214 1024  377   10.306   -0.263   1.027
+213.136.0.252   .PPS.            1 u  745 1024  377    6.285   -0.110   0.578
 10.0.5.3        134.130.4.17     2 s  874 1024  376    0.181   -0.022   0.289

----------

% ntpq -4 -n -c rl -wp localhost
associd=0 status=0615 leap_none, sync_ntp, 1 event, clock_sync,
version="ntpd 4.2.8p12-a (1)", processor="amd64",
system="FreeBSD/11.2-STABLE", leap=00, stratum=2, precision=-24,
rootdelay=10.842, rootdisp=27.292, refid=192.53.103.103,
reftime=e1a36434.8416e548  Tue, Dec 17 2019 15:27:00.515,
clock=e1a36735.530754d7  Tue, Dec 17 2019 15:39:49.324, peer=31263,
tc=10, mintc=3, offset=0.054371, frequency=73.858, sys_jitter=0.039632,
clk_jitter=0.075, clk_wander=0.005, tai=37, leapsec=201701010000,
expire=202006280000

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
-134.130.4.17    .MRS.            1 u  747 1024  377    2.697   -0.008   4.593
+134.130.5.17    .MRS.            1 u  159 1024  377    2.704    0.005   3.223
-129.143.2.23    .GPS.            1 u  642 1024  377    7.911   -0.033   0.441
-129.143.2.33    .PZF.            1 u  312 1024  377    7.862   -0.325   0.061
+192.87.106.2    .GPS.            1 u  949 1024  377    6.713    0.119   0.065
-192.53.103.108  .PTB.            1 u  872 1024  377   14.247    1.696   7.276
*192.53.103.103  .PTB.            1 u  769 1024  377   10.842    0.035   0.303
-213.136.0.252   .PPS.            1 u  733 1024  377    6.967    0.038   0.078
 10.0.32.249     213.136.0.252    2 s  935 1024  376    0.120   -0.066   0.103
Comment 73 Vinícius Zavam freebsd_committer 2019-12-17 15:05:37 UTC
(In reply to Cy Schubert from comment #70)

2 of the servers you see are in the same VLAN; there's also no firewall filtering this kind of thing (not even scrubbing udp/123) for the other 2.

you can check the 'delay' column, but anyway ... 

% ping -c 10 10.0.5.3 | grep -B1 avg
10 packets transmitted, 10 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.183/0.402/1.267/0.309 ms

% ping -c 10 10.0.32.248 | grep -B1 avg
10 packets transmitted, 10 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.958/1.024/1.157/0.064 ms
Comment 74 Cy Schubert freebsd_committer 2019-12-17 20:29:52 UTC
(In reply to Vinícius Zavam from comment #71)
Don't delete the file, just clear it (as in empty it).
Comment 75 Cy Schubert freebsd_committer 2019-12-17 20:31:07 UTC
(In reply to Vinícius Zavam from comment #72)
Can we see the ntpd.drift file for the two servers that do work, please?
Comment 76 Cy Schubert freebsd_committer 2019-12-17 20:34:39 UTC
(In reply to Vinícius Zavam from comment #71)
Did you run ntpdate prior to starting ntpd?
Comment 77 Cy Schubert freebsd_committer 2019-12-17 20:35:48 UTC
Let me ask again, please. What is in rc.conf? Can you list it, please?
Comment 78 Harlan Stenn 2019-12-17 21:48:28 UTC
(In reply to Cy Schubert from comment #76)

I'm curious about this one, Cy.  This used to be recommended practice before we had the -g option to ntpd, but we're 95%+ comfortable saying that it's no longer needed.  I've had one person say they know cases where it matters, but I haven't seen the supporting information yet.
Comment 79 Harlan Stenn 2019-12-17 21:55:46 UTC
(In reply to Cy Schubert from comment #70)
That's not a large offset/jitter given that only 3 or 4 polls have occurred.

Look at the 'reach' indicator.  7 means only 3 polls have happened, and 17 means only 4 polls have happened.

I'm betting the 'iburst' flag is not being provided.

We recommend using 'iburst' because it will sync your clock sooner.
Comment 80 Harlan Stenn 2019-12-17 21:58:33 UTC
(In reply to Harlan Stenn from comment #79)
If you *are* using iburst, then you're running 'ntpq' about 5 seconds after starting ntpd.  Wait at least 15 seconds.  iburst sends up to 8 packets at 2 second intervals.  It takes at least 5 samples for a warm-start to settle, which usually happens in about 11 seconds' time.
Comment 81 Cy Schubert freebsd_committer 2019-12-18 02:18:29 UTC
(In reply to Harlan Stenn from comment #78)
You are correct Harlan. -g will work.
Comment 82 Cy Schubert freebsd_committer 2019-12-18 02:20:13 UTC
(In reply to Harlan Stenn from comment #80)
Yes, he's specified iburst on:

pool 0.freebsd.pool.ntp.org iburst
Comment 83 Vinícius Zavam freebsd_committer 2019-12-23 09:36:28 UTC
(In reply to Cy Schubert from comment #75)

egypcio@ntp1:~ % uptime
10:33AM  up 5 days, 18:13, 4 users, load averages: 0.15, 0.14, 0.15
egypcio@ntp1:~ % cat /var/db/ntpd.drift
23.727

egypcio@ntp2:~ % uptime
10:33AM  up 6 days, 18:30, 3 users, load averages: 0.12, 0.11, 0.09
egypcio@ntp2:~ % cat /var/db/ntpd.drift
24.216
Comment 84 Vinícius Zavam freebsd_committer 2019-12-23 09:37:36 UTC
(In reply to Cy Schubert from comment #76)

yes, I did. as we see on Comment 69 this was the command line I ran:

# date ; ntpq -4 -n -c rl -wp localhost ; service ntpd stop ; rm /var/db/ntp/ntpd.drift ; ntpdate ntp2.prod.localdomain.de ; service ntpd start ; sleep 1800 ; date ; ntpq -4 -n -c rl -wp localhost
Comment 85 Vinícius Zavam freebsd_committer 2019-12-23 09:42:10 UTC
(In reply to Cy Schubert from comment #74)

the ntpd creates a new one anyway (that's what I did see on its manpage). should not bother. right?

btw, here is the current status of the drift file on the stable/12:

egypcio@stable12:~ % ls -lash /var/db/ntp/ntpd.pid 
4 -rw-r--r--  1 ntpd  ntpd     5B Dec 17 14:49 /var/db/ntp/ntpd.pid

egypcio@stable12:~ % cat /var/db/ntp/ntpd.drift
500.000
Comment 86 Vinícius Zavam freebsd_committer 2019-12-23 09:49:58 UTC
(In reply to Cy Schubert from comment #77)

here is the rc.conf on the stable12 machine:

keymap="de"
keyrate="fast"
saver="blank"
hostname="stable12.prod.localdomain.de"
cloned_interfaces="gif21 gif22 gif31 gif32"
vlans_bge0="vlan21 vlan22 vlan23 vlan24 vlan249"
vlans_bge1="vlan3011"
create_args_vlan21="vlan 21"
create_args_vlan22="vlan 22"
create_args_vlan23="vlan 23"
create_args_vlan24="vlan 24"
create_args_vlan249="vlan 249"
create_args_vlan3011="vlan 3011"
ifconfig_bge0="inet 10.0.5.7/24 -lro -tso -rxcsum -txcsum"
ifconfig_bge1="up -lro -tso -rxcsum -txcsum"
ifconfig_enc0="up"
ifconfig_vlan21="inet 10.0.21.251/24"
ifconfig_vlan22="inet 10.0.22.251/24"
ifconfig_vlan23="inet 10.0.23.251/24"
ifconfig_vlan24="inet 10.0.24.251/24"
ifconfig_vlan249="inet 10.0.249.5/29"
ifconfig_vlan3011="inet 192.168.123.12/24"
. /etc/rc.conf.carp.vlan3011
ifconfig_bge0_alias0="inet    10.0.5.254/32  vhid 210 pass jurubeba advskew 232"
ifconfig_vlan21_alias0="inet  10.0.21.254/32 vhid 21  pass jurubeba advskew 232"
ifconfig_vlan22_alias0="inet  10.0.22.254/32 vhid 22  pass jurubeba advskew 232"
ifconfig_vlan23_alias0="inet  10.0.23.253/32 vhid 23  pass jurubeba advskew 232"
ifconfig_vlan24_alias0="inet  10.0.24.254/32 vhid 24  pass jurubeba advskew 232"
ifconfig_vlan249_alias0="inet 10.0.249.1/29  vhid 250 pass jurubeba advskew 232"
ifconfig_gif21="10.0.248.26/30 10.0.248.25 tunnel 192.168.123.12 192.168.99.59 mtu 1416 description 'IPSec VPN to abc2a'"
ifconfig_gif22="10.0.248.10/30 10.0.248.9  tunnel 192.168.123.12 172.16.26.52  mtu 1416 description 'IPSec VPN to abc2b'"
ifconfig_gif32="10.0.248.46/30 10.0.248.45 tunnel 192.168.123.12 172.31.41.8   mtu 1416 description 'IPSec VPN to qwerty2a'"
gateway_enable="YES"
defaultrouter="192.168.123.1"
ipv6_activate_all_interfaces="YES"
ipv6_gateway_enable="YES"
ifconfig_bge0_ipv6="inet6 fd00:0:0:5::7/64"
ifconfig_bge0_alias1="inet6 fd00:0:0:5::254/64 vhid 210"
ifconfig_vlan249_ipv6="inet6 fd00:0:0:249::5/64"
ifconfig_vlan249_alias1="inet6 fd00:0:0:249::1/64 vhid 250"
ifconfig_vlan3011_ipv6="inet6 2001:db8:fb5d::12/64"
ipv6_defaultrouter="fe80::1%vlan3011"
powerd_enable="YES"
powerd_flags="-a maximum -b hiadaptive"
ntpdate_enable="YES"
ntpd_enable="YES"
ntpd_flags="${ntpd_flags} -4 --logfile=/var/log/ntpd.log --statsdir=/etc/ntp/stats"
ntpd_sync_on_start="YES"
sshd_enable="YES"
syslogd_flags="-s -4"
postfix_enable="YES"
sendmail_enable="NONE"
sendmail_msp_queue_enable="NO"
sendmail_outbound_enable="NO"
sendmail_submit_enable="NO"
quagga_enable="YES"
quagga_daemons="zebra ospfd"
pf_enable="YES"
pflog_enable="YES"
pfsync_enable="YES"
pfsync_syncdev="vlan249"
pfsync_syncpeer="10.0.249.3"
ipsec_enable="YES"
racoon_enable="YES"
racoon_flags="-l /var/log/racoon.log"
ossec_hids_enable="YES" 
zabbix_agentd_enable="YES"
nrpe3_enable="YES"
Comment 87 Vinícius Zavam freebsd_committer 2019-12-23 09:55:48 UTC
I left the live image running for about  5 days, and here are the results:

root@head-r355121-amd64:~ # uptime
 9:50AM  up 5 days, 19:56, 2 users, load averages: 0.09, 0.22, 0.23

root@head-r355121-amd64:~ # ps afxuww | grep ntp
root   3067   0.0  0.0 19308 6352  -  Ss   Tue13       0:22.59 /usr/sbin/ntpd --logfile=/var/log/ntpd.log --statsdir=/var/db/ntp/stats -p /var/db/ntp/ntpd.pid -c /var/etc_ntp.conf -f /var/db/ntp/ntpd.drift -4 -g
root  16950   0.0  0.0 11364 2620  0  S+   09:50       0:00.00 grep ntp

root@head-r355121-amd64:~ # cat /var/db/ntp/ntpd.drift
500.000

root@head-r355121-amd64:~ # ntpq -4 -n -c rl -wp localhost
associd=0 status=c613 leap_alarm, sync_ntp, 1 event, spike_detect,
version="ntpd 4.2.8p12-a (1)", processor="amd64",
system="FreeBSD/13.0-CURRENT", leap=11, stratum=3, precision=-23,
rootdelay=23.848, rootdisp=1548.403, refid=80.237.128.148,
reftime=e1ab0c2b.f6e024b2  Mon, Dec 23 2019  9:49:31.964,
clock=e1ab0c60.d73bbaf7  Mon, Dec 23 2019  9:50:24.840, peer=28700, tc=6,
mintc=3, offset=0.000000, frequency=500.000, sys_jitter=87.612594,
clk_jitter=0.000, clk_wander=0.000, leapsec=201701010000,
expire=201912280000

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 0.freebsd.pool.ntp.org
                 .POOL.          16 p    -   64    0    0.000    0.000   0.000
 83.68.137.76    131.130.251.107  2 u   53   64    1   16.140  323.987 192.995
 5.39.184.5      46.243.26.34     2 u   48   64    1    7.025  321.914 201.789
 212.18.3.19     212.18.1.106     2 u   16   64    1   10.195  322.169 252.769
 82.64.45.50     .GPS.            1 u   15   64    1   32.063  541.354 140.598
 88.99.216.194   131.188.3.223    2 u   15   64    1    8.995  659.318 214.715
 82.100.248.10   131.188.3.222    2 u   50   64    1    4.476  481.363 114.111
 162.159.200.123 10.48.8.4        3 u   48   64    1    1.917  594.497 184.049
 145.239.0.227   131.188.3.223    2 u   35   64    1    9.819  621.196 194.087
 162.159.200.1   10.48.8.4        3 u   46   64    1    1.897  485.164 116.870
 116.203.149.9   131.188.3.222    2 u   35   64    1    7.630  504.075 123.998
*80.237.128.148  162.23.41.10     2 u   53   64    1    7.964  587.071 182.109
 144.76.60.190   131.188.3.221    2 u   24   64    1    9.123  638.931 208.746

root@head-r355121-amd64:~ # cat /etc/rc.conf
sendmail_enable="NONE"
hostid_enable="NO"

root@head-r355121-amd64:~ # grep -v ^# /var/etc_ntp.conf
leapfile "/var/db/ntpd.leap-seconds.list"
pool 0.freebsd.pool.ntp.org iburst
restrict 127.0.0.1
restrict ::1
restrict default limited kod nomodify notrap noquery nopeer
restrict source  limited kod nomodify notrap noquery
statistics clockstats cryptostats loopstats peerstats rawstats sysstats
tos minclock 3 maxclock 6

ntpd.log -> https://share.riseup.net/#QaFZ3JaDeJcEL3AX6gla9Q
Comment 88 Vinícius Zavam freebsd_committer 2019-12-23 10:01:06 UTC
(In reply to Vinícius Zavam from comment #83)

quering the status and stuff of the 2 working servers (running 12.1R-p1):

% ntpq -4 -n -c rl -wp ntp1
associd=0 status=0615 leap_none, sync_ntp, 1 event, clock_sync,
version="ntpd 4.2.8p12-a (1)", processor="amd64",
system="FreeBSD/12.1-RELEASE-p1", leap=00, stratum=2, precision=-24,
rootdelay=5.898, rootdisp=17.291, refid=213.136.0.252,
reftime=e1ab0bdd.8195ba5d  Mon, Dec 23 2019  9:48:13.506,
clock=e1ab0e6a.e53c3b38  Mon, Dec 23 2019  9:59:06.895, peer=37729, tc=8,
mintc=3, offset=0.362365, frequency=23.796, sys_jitter=0.051579,
clk_jitter=0.140, clk_wander=0.020, tai=37, leapsec=201701010000,
expire=202006280000

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
-134.130.4.17    .MRS.            1 u  164  256  377    2.231    0.337   0.187
-134.130.5.17    .MRS.            1 u  232  256  377    2.212    0.501   1.212
-129.143.2.23    .GPS.            1 u  112  256  377    7.405    0.283   0.215
-129.143.2.33    .PZF.            1 u  131  256  377    7.482   -0.020   1.797
-192.87.106.2    .GPS.            1 u  242  256  377    5.972    0.664   0.128
+192.53.103.108  .PTB.            1 u  188  256  377   10.362    0.302   1.297
+192.53.103.103  .PTB.            1 u  127  256  377   10.424    0.335   1.398
*213.136.0.252   .PPS.            1 u  121  256  377    5.898    0.382   0.113
 10.0.5.4        134.130.4.17     2 s  174  256  376    0.138    0.582   0.082

----------

% ntpq -4 -n -c rl -wp ntp2
associd=0 status=0618 leap_none, sync_ntp, 1 event, no_sys_peer,
version="ntpd 4.2.8p12-a (1)", processor="amd64",
system="FreeBSD/12.1-RELEASE-p1", leap=00, stratum=2, precision=-24,
rootdelay=2.515, rootdisp=19.147, refid=134.130.4.17,
reftime=e1ab0c8c.c0f35810  Mon, Dec 23 2019  9:51:08.753,
clock=e1ab0e71.f26232e3  Mon, Dec 23 2019  9:59:13.946, peer=41863,
tc=10, mintc=3, offset=-0.054842, frequency=24.215, sys_jitter=0.118919,
clk_jitter=0.077, clk_wander=0.011, tai=37, leapsec=201701010000,
expire=202006280000

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*134.130.4.17    .MRS.            1 u  485  512  377    2.515   -0.057   0.121
-134.130.5.17    .MRS.            1 u  139 1024  377    2.369   -0.208   0.987
-129.143.2.23    .GPS.            1 u  983 1024  277    7.655   -0.276   0.073
-129.143.2.33    .PZF.            1 u   79 1024  377    7.541   -0.425   0.083
+192.87.106.2    .GPS.            1 u  235 1024  377    5.548   -0.080   0.110
-192.53.103.108  .PTB.            1 u  270 1024  377   10.481   -0.180   0.885
-192.53.103.103  .PTB.            1 u  298 1024  377   10.411   -0.177   1.438
+213.136.0.252   .PPS.            1 u  355 1024  377    6.282   -0.030   0.195
 10.0.5.3        213.136.0.252    2 s  172  256  377    0.144   -0.585   0.084
Comment 89 Vinícius Zavam freebsd_committer 2019-12-23 10:02:51 UTC
(In reply to Cy Schubert from comment #81)

which is being used already
Comment 90 Vinícius Zavam freebsd_committer 2019-12-23 10:04:56 UTC
(In reply to Vinícius Zavam from comment #85)

a bit more from the stable/12 machine:

% ntpq -4 -n -c rl localhost
associd=0 status=c018 leap_alarm, sync_unspec, 1 event, no_sys_peer,
version="ntpd 4.2.8p12-a (1)", processor="amd64",
system="FreeBSD/12.1-STABLE", leap=11, stratum=16, precision=-22,
rootdelay=0.000, rootdisp=1.020, refid=STEP, reftime=(no time),
clock=e1ab0f74.8d96c57d  Mon, Dec 23 2019 10:03:32.553, peer=0, tc=6,
mintc=3, offset=0.000000, frequency=500.000, sys_jitter=0.000238,
clk_jitter=0.000, clk_wander=0.000, tai=37, leapsec=201701010000,
expire=202006280000
Comment 91 Vinícius Zavam freebsd_committer 2019-12-23 10:11:01 UTC
(In reply to Vinícius Zavam from comment #90)

https://imgur.com/a/6p1tTue
Comment 92 Cy Schubert freebsd_committer 2020-01-05 00:40:16 UTC
What is the output of ntptime?
Comment 93 Vinícius Zavam freebsd_committer 2020-01-06 09:25:49 UTC
(In reply to Cy Schubert from comment #92)

root@freebsd-HEAD-r356310-amd64:~ # ntptime
ntp_gettime() returns code 5 (ERROR)
  time e1bd7b28.09d56000  Mon, Jan  6 2020  9:23:52.038, (.690038412),
  maximum error 16453500 us, estimated error 16000000 us, TAI offset 0
ntp_adjtime() returns code 5 (ERROR)
  modes 0x0 (),
  offset 0.000 us, frequency 500.000 ppm, interval 1 s,
  maximum error 16453500 us, estimated error 16000000 us,
  status 0x41 (PLL,UNSYNC),
  time constant 3, precision 0.000 us, tolerance 496 ppm,

root@freebsd-HEAD-r356310-amd64:~ # ps afxuww | grep ntp
root  3786   0.0  0.0 19108 6416  -  Ss   09:08     0:00.07 /usr/sbin/ntpd --logfile=/var/log/ntpd.log --statsdir=/var/db/ntp/stats -p /var/db/ntp/ntpd.pid -c /var/etc_ntp.conf -f /var/db/ntp/ntpd.drift -4 -g
root  3824   0.0  0.0 11376 2632  0  S+   09:23     0:00.00 grep ntp

root@freebsd-HEAD-r356310-amd64:~ # ntpq -4 -n -c rl -wp localhost
associd=0 status=0613 leap_none, sync_ntp, 1 event, spike_detect,
version="ntpd 4.2.8p12-a (1)", processor="amd64",
system="FreeBSD/13.0-CURRENT", leap=00, stratum=3, precision=-23,
rootdelay=26.166, rootdisp=537.015, refid=5.199.135.170,
reftime=e1bd7ae3.9a3374c2  Mon, Jan  6 2020  9:22:43.602,
clock=e1bd7b01.85243efb  Mon, Jan  6 2020  9:23:13.520, peer=46567, tc=6,
mintc=3, offset=73.042339, frequency=500.000, sys_jitter=52.538034,
clk_jitter=25.824, clk_wander=0.000, tai=37, leapsec=201701010000,
expire=202006280000

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 0.freebsd.pool.ntp.org
                 .POOL.          16 p    -   64    0    0.000    0.000   0.000
 85.25.148.4     141.2.22.74      4 u    7   64    1    7.366  331.053 183.178
 88.99.216.194   131.188.3.223    2 u    9   64    1    9.158  326.029 176.823
 213.239.239.166 205.46.178.169   2 u    8   64    1   18.858  321.180 176.054
 212.18.3.18     212.18.1.106     2 u    5   64    1   10.313  331.719 178.953
+159.69.150.81   131.188.3.220    2 u   44   64    1    7.864  259.263 145.789
+88.198.17.248   212.18.3.18      3 u   30   64    1   24.399  277.903 190.095
+85.220.190.246  213.172.96.14    2 u   28   64    1    4.689  289.230 201.493
+78.46.90.21     129.69.1.153     2 u   29   64    1    9.562  287.327 201.722
*5.199.135.170   131.188.3.222    2 u   30   64    1   17.117  277.570 188.922
+116.203.151.74  131.188.3.222    2 u   29   64    1    7.700  287.364 195.871
+176.9.1.211     122.227.206.195  3 u   28   64    1    9.144  288.973 201.673
+62.116.130.3    192.53.103.104   2 u   85   64    1    9.756  185.961 114.962

root@freebsd-HEAD-r356310-amd64:~ # uname -rmpsiUK
FreeBSD 13.0-CURRENT amd64 amd64 GENERIC 1300073 1300073

root@freebsd-HEAD-r356310-amd64:~ # ll /var/db/ntp/ntpd.pid
-rw-r--r--  1 ntpd  ntpd  4 Jan  6 09:08 /var/db/ntp/ntpd.pid

root@freebsd-HEAD-r356310-amd64:~ # date
Mon Jan  6 09:23:26 UTC 2020

root@freebsd-HEAD-r356310-amd64:~ # uptime
 9:23AM  up 28 mins, 2 users, load averages: 0.17, 0.19, 0.16
Comment 94 Vinícius Zavam freebsd_committer 2020-01-06 13:22:07 UTC
(In reply to Vinícius Zavam from comment #93)

here, 'ntptime' output from my ntp1 and ntp2 boxes (12.1-Rp1):

egypcio@ntp1:~ % ntptime
ntp_gettime() returns code 0 (OK)
  time e1bdb268.5d6c508c  Mon, Jan  6 2020 14:19:36.364, (.364934606),
  maximum error 240727 us, estimated error 192 us, TAI offset 37
ntp_adjtime() returns code 0 (OK)
  modes 0x0 (),
  offset -1501.932 us, frequency 23.589 ppm, interval 1 s,
  maximum error 240727 us, estimated error 192 us,
  status 0x2001 (PLL,NANO),
  time constant 9, precision 1.000 us, tolerance 496 ppm,

egypcio@ntp2:~ % ntptime
ntp_gettime() returns code 0 (OK)
  time e1bdb26c.9baf6408  Mon, Jan  6 2020 14:19:40.608, (.608145871),
  maximum error 114455 us, estimated error 84 us, TAI offset 37
ntp_adjtime() returns code 0 (OK)
  modes 0x0 (),
  offset -382.554 us, frequency 24.064 ppm, interval 1 s,
  maximum error 114455 us, estimated error 84 us,
  status 0x2001 (PLL,NANO),
  time constant 8, precision 1.000 us, tolerance 496 ppm,
Comment 95 Vinícius Zavam freebsd_committer 2020-01-06 13:25:07 UTC
silly question: as this issue definitely seems to also be influenced by the hardware we are using, and not only the FreeBSD version/revision, what other information would you need to help debugging it? - I mentioned the CPU and also server model before.
Comment 96 Cy Schubert freebsd_committer 2020-01-06 13:59:49 UTC
Yes, ntptime tells us this.

On only the affected system,

sysctl -a | egrep 'timecounter|eventtimer'
Comment 97 Vinícius Zavam freebsd_committer 2020-01-06 16:49:27 UTC
(In reply to Cy Schubert from comment #96)

root@head-r356310-amd64:~ # sysctl -a | egrep 'timecounter|eventtimer'
kern.timecounter.tsc_shift: 1
kern.timecounter.smp_tsc_adjust: 0
kern.timecounter.smp_tsc: 1
kern.timecounter.invariant_tsc: 1
kern.timecounter.fast_gettime: 1
kern.timecounter.tick: 1
kern.timecounter.choice: ACPI-fast(900) HPET(950) i8254(0) TSC(1000) dummy(-1000000)
kern.timecounter.hardware: TSC
kern.timecounter.alloweddeviation: 5
kern.timecounter.timehands_count: 2
kern.timecounter.stepwarnings: 0
kern.timecounter.tc.ACPI-fast.quality: 900
kern.timecounter.tc.ACPI-fast.frequency: 3579545
kern.timecounter.tc.ACPI-fast.counter: 16329298
kern.timecounter.tc.ACPI-fast.mask: 16777215
kern.timecounter.tc.HPET.quality: 950
kern.timecounter.tc.HPET.frequency: 24000000
kern.timecounter.tc.HPET.counter: 1316611790
kern.timecounter.tc.HPET.mask: 4294967295
kern.timecounter.tc.i8254.quality: 0
kern.timecounter.tc.i8254.frequency: 1193182
kern.timecounter.tc.i8254.counter: 45741
kern.timecounter.tc.i8254.mask: 65535
kern.timecounter.tc.TSC.quality: 1000
kern.timecounter.tc.TSC.frequency: 1900000000
kern.timecounter.tc.TSC.counter: 307319750
kern.timecounter.tc.TSC.mask: 4294967295
kern.eventtimer.periodic: 0
kern.eventtimer.timer: LAPIC
kern.eventtimer.idletick: 0
kern.eventtimer.singlemul: 2
kern.eventtimer.choice: LAPIC(600) HPET(550) HPET1(440) HPET2(440) i8254(100) RTC(0)
kern.eventtimer.et.HPET2.quality: 440
kern.eventtimer.et.HPET2.frequency: 24000000
kern.eventtimer.et.HPET2.flags: 3
kern.eventtimer.et.HPET1.quality: 440
kern.eventtimer.et.HPET1.frequency: 24000000
kern.eventtimer.et.HPET1.flags: 3
kern.eventtimer.et.HPET.quality: 550
kern.eventtimer.et.HPET.frequency: 24000000
kern.eventtimer.et.HPET.flags: 7
kern.eventtimer.et.i8254.quality: 100
kern.eventtimer.et.i8254.frequency: 1193182
kern.eventtimer.et.i8254.flags: 1
kern.eventtimer.et.RTC.quality: 0
kern.eventtimer.et.RTC.frequency: 32768
kern.eventtimer.et.RTC.flags: 17
kern.eventtimer.et.LAPIC.quality: 600
kern.eventtimer.et.LAPIC.frequency: 1900000000
kern.eventtimer.et.LAPIC.flags: 7
Comment 98 Vinícius Zavam freebsd_committer 2020-01-07 09:47:16 UTC
Created attachment 210496 [details]
ntpdate_offsetvars.txt

I left the machine running (for about 16h) with a little script running in background to check the offset variation of its clock "ticking" WITHOUT RUNNING `ntpd`.

here is the code:

#!/bin/sh
while true; do
 ntpdate -qp1 ntp1.netcologne.de
 sleep 60
done

attached you have the output.
Comment 99 Cy Schubert freebsd_committer 2020-01-07 13:43:10 UTC
What is the output of,

sysctl dev.cpu.0.cx_lowest
Comment 100 Cy Schubert freebsd_committer 2020-01-07 13:44:59 UTC
Actually, sysctl -a | grep cx_lowest
Comment 101 Vinícius Zavam freebsd_committer 2020-01-07 13:52:01 UTC
(In reply to Cy Schubert from comment #100)

% sysctl -a | grep cx_lowest
hw.acpi.cpu.cx_lowest: C1
dev.cpu.5.cx_lowest: C1
dev.cpu.4.cx_lowest: C1
dev.cpu.3.cx_lowest: C1
dev.cpu.2.cx_lowest: C1
dev.cpu.1.cx_lowest: C1
dev.cpu.0.cx_lowest: C1
Comment 102 Cy Schubert freebsd_committer 2020-01-07 13:56:07 UTC
Can you rebuild your kernel without EARLY_AP_STARTUP ?
Comment 103 Vinícius Zavam freebsd_committer 2020-01-07 14:06:35 UTC
(In reply to Cy Schubert from comment #102)

sure! I should start building in a few, again I will use the 'release.sh' and get a bootable USB image to try it out - will take a while.
Comment 104 Vinícius Zavam freebsd_committer 2020-01-07 14:08:24 UTC
(In reply to Vinícius Zavam from comment #103)

# /usr/src/sys/amd64/conf/PR242137 (HEAD@r356428)
include         GENERIC
ident           PR242137
nooptions       EARLY_AP_STARTUP
Comment 105 Vinícius Zavam freebsd_committer 2020-01-08 11:28:31 UTC
(In reply to Vinícius Zavam from comment #104)

root@head-r356428-amd64:~ # uptime
11:24AM  up 49 mins, 2 users, load averages: 0.15, 0.20, 0.21
root@head-r356428-amd64:~ # ps afxuww | grep ntp
root  3287   0.0  0.0 19136 6420  -  Ss   10:55     0:00.10 /usr/sbin/ntpd --logfile=/var/log/ntpd.log --statsdir=/var/db/ntp/stats -p /var/db/ntp/ntpd.pid -c /var/etc_ntp.conf -f /var/db/ntp/ntpd.drift -4 -g
root  3321   0.0  0.0 11376 2600  0  S+   11:24     0:00.00 grep ntp



root@head-r356428-amd64:~ # ntpq -4 -n -c rl -wp localhost
associd=0 status=c613 leap_alarm, sync_ntp, 1 event, spike_detect,
version="ntpd 4.2.8p12-a (1)", processor="amd64",
system="FreeBSD/13.0-CURRENT", leap=11, stratum=3, precision=-23,
rootdelay=14.370, rootdisp=1525.451, refid=88.99.76.254,
reftime=e1c03a04.fd3def83  Wed, Jan  8 2020 11:22:44.989,
clock=e1c03a8d.1afc38e7  Wed, Jan  8 2020 11:25:01.105, peer=40336, tc=6,
mintc=3, offset=0.000000, frequency=500.000, sys_jitter=93.026842,
clk_jitter=0.000, clk_wander=0.000, tai=37, leapsec=201701010000,
expire=202006280000

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 0.freebsd.pool.ntp.org
                 .POOL.          16 p    -   64    0    0.000    0.000   0.000
 131.188.3.222   .MBGh.           1 u   45   64    1    7.803  442.385 261.983
 144.76.76.107   205.46.178.169   2 u   46   64    1    8.998  444.277 258.226
 162.159.200.123 10.48.8.4        3 u   43   64    1    1.893  443.045 265.364
 159.69.150.81   131.188.3.220    2 u   41   64    1    7.715  838.467 260.863
 80.127.152.30   131.211.8.244    2 u   30   64    1    9.257  855.442 268.080
 91.209.0.17     232.6.188.111    2 u   16   64    1   17.080  657.616 134.005
 91.220.110.117  236.118.239.31   2 u   13   64    1   56.410  773.421 192.055
 195.219.205.9   195.219.14.21    2 u   14   64    1   15.401  440.598 315.054
+178.63.247.119  17.253.54.123    2 u   13   64    1    9.463  773.580 217.357
*88.99.76.254    17.253.54.125    2 u   12   64    1    9.289  545.807 193.492
 193.141.27.6    192.53.103.108   2 u   10   64    1    6.720  551.751 196.067
 141.30.228.4    212.18.3.18      3 u    9   64    1   20.020  445.637 276.813



root@head-r356428-amd64:~ # ntptime 
ntp_gettime() returns code 0 (OK)
  time e1c03a8e.e54c9860  Wed, Jan  8 2020 11:25:02.895, (.895700329),
  maximum error 925331 us, estimated error 0 us, TAI offset 37
ntp_adjtime() returns code 0 (OK)
  modes 0x0 (),
  offset 0.000 us, frequency 500.000 ppm, interval 1 s,
  maximum error 925331 us, estimated error 0 us,
  status 0x2001 (PLL,NANO),
  time constant 6, precision 1.000 us, tolerance 496 ppm,



root@head-r356428-amd64:~ # sysctl -a | grep cx_lowest
hw.acpi.cpu.cx_lowest: C1
dev.cpu.5.cx_lowest: C1
dev.cpu.4.cx_lowest: C1
dev.cpu.3.cx_lowest: C1
dev.cpu.2.cx_lowest: C1
dev.cpu.1.cx_lowest: C1
dev.cpu.0.cx_lowest: C1



root@head-r356428-amd64:~ # sysctl -a | egrep 'timecounter|eventtimer'
kern.timecounter.tsc_shift: 1
kern.timecounter.smp_tsc_adjust: 0
kern.timecounter.smp_tsc: 1
kern.timecounter.invariant_tsc: 1
kern.timecounter.fast_gettime: 1
kern.timecounter.tick: 1
kern.timecounter.choice: TSC(1000) ACPI-fast(900) HPET(950) i8254(0) dummy(-1000000)
kern.timecounter.hardware: TSC
kern.timecounter.alloweddeviation: 5
kern.timecounter.timehands_count: 2
kern.timecounter.stepwarnings: 0
kern.timecounter.tc.TSC.quality: 1000
kern.timecounter.tc.TSC.frequency: 1900000000
kern.timecounter.tc.TSC.counter: 4137183290
kern.timecounter.tc.TSC.mask: 4294967295
kern.timecounter.tc.ACPI-fast.quality: 900
kern.timecounter.tc.ACPI-fast.frequency: 3579545
kern.timecounter.tc.ACPI-fast.counter: 11808359
kern.timecounter.tc.ACPI-fast.mask: 16777215
kern.timecounter.tc.HPET.quality: 950
kern.timecounter.tc.HPET.frequency: 24000000
kern.timecounter.tc.HPET.counter: 283868598
kern.timecounter.tc.HPET.mask: 4294967295
kern.timecounter.tc.i8254.quality: 0
kern.timecounter.tc.i8254.frequency: 1193182
kern.timecounter.tc.i8254.counter: 38880
kern.timecounter.tc.i8254.mask: 65535
kern.eventtimer.periodic: 0
kern.eventtimer.timer: LAPIC
kern.eventtimer.idletick: 0
kern.eventtimer.singlemul: 2
kern.eventtimer.choice: LAPIC(600) HPET(550) HPET1(440) HPET2(440) i8254(100) RTC(0)
kern.eventtimer.et.HPET2.quality: 440
kern.eventtimer.et.HPET2.frequency: 24000000
kern.eventtimer.et.HPET2.flags: 3
kern.eventtimer.et.HPET1.quality: 440
kern.eventtimer.et.HPET1.frequency: 24000000
kern.eventtimer.et.HPET1.flags: 3
kern.eventtimer.et.HPET.quality: 550
kern.eventtimer.et.HPET.frequency: 24000000
kern.eventtimer.et.HPET.flags: 7
kern.eventtimer.et.i8254.quality: 100
kern.eventtimer.et.i8254.frequency: 1193182
kern.eventtimer.et.i8254.flags: 1
kern.eventtimer.et.RTC.quality: 0
kern.eventtimer.et.RTC.frequency: 32768
kern.eventtimer.et.RTC.flags: 17
kern.eventtimer.et.LAPIC.quality: 600
kern.eventtimer.et.LAPIC.frequency: 1900000000
kern.eventtimer.et.LAPIC.flags: 7
Comment 106 Vinícius Zavam freebsd_committer 2020-01-08 13:03:35 UTC
(In reply to Vinícius Zavam from comment #98)

I ran this test again, using ***12.1-RELEASE*** and here is the output of few queries:


-----
server 81.173.193.222, stratum 2, offset 0.723093, delay 0.02759
 8 Jan 12:43:56 ntpdate[3096]: step time server 81.173.193.222 offset 0.723093 sec
server 81.173.193.222, stratum 2, offset 0.726101, delay 0.02786
 8 Jan 12:44:56 ntpdate[3103]: step time server 81.173.193.222 offset 0.726101 sec
server 81.173.193.222, stratum 2, offset 0.729273, delay 0.02727
 8 Jan 12:45:56 ntpdate[3105]: step time server 81.173.193.222 offset 0.729273 sec
server 81.173.193.222, stratum 2, offset 0.732274, delay 0.02727
 8 Jan 12:46:56 ntpdate[3107]: step time server 81.173.193.222 offset 0.732274 sec
server 81.173.193.222, stratum 2, offset 0.734884, delay 0.02863
 8 Jan 12:47:56 ntpdate[3110]: step time server 81.173.193.222 offset 0.734884 sec
server 81.173.193.222, stratum 2, offset 0.738171, delay 0.02786
 8 Jan 12:48:56 ntpdate[3113]: step time server 81.173.193.222 offset 0.738171 sec
server 81.173.193.222, stratum 2, offset 0.741373, delay 0.02745
 8 Jan 12:49:57 ntpdate[3115]: step time server 81.173.193.222 offset 0.741373 sec
server 81.173.193.222, stratum 2, offset 0.742197, delay 0.03177
 8 Jan 12:50:57 ntpdate[3117]: step time server 81.173.193.222 offset 0.742197 sec
server 81.173.193.222, stratum 2, offset 0.747455, delay 0.02739
 8 Jan 12:51:57 ntpdate[3119]: step time server 81.173.193.222 offset 0.747455 sec
server 81.173.193.222, stratum 2, offset 0.750519, delay 0.02797
 8 Jan 12:52:57 ntpdate[3121]: step time server 81.173.193.222 offset 0.750519 sec
server 81.173.193.222, stratum 2, offset 0.753328, delay 0.02788
 8 Jan 12:53:57 ntpdate[3123]: step time server 81.173.193.222 offset 0.753328 sec
server 81.173.193.222, stratum 2, offset 0.756456, delay 0.02759
 8 Jan 12:54:58 ntpdate[3125]: step time server 81.173.193.222 offset 0.756456 sec
server 81.173.193.222, stratum 2, offset 0.759400, delay 0.02742
 8 Jan 12:55:58 ntpdate[3132]: step time server 81.173.193.222 offset 0.759400 sec
server 81.173.193.222, stratum 2, offset 0.762648, delay 0.02757
 8 Jan 12:56:58 ntpdate[3134]: step time server 81.173.193.222 offset 0.762648 sec
server 81.173.193.222, stratum 2, offset 0.765475, delay 0.02742
 8 Jan 12:57:58 ntpdate[3136]: step time server 81.173.193.222 offset 0.765475 sec
server 81.173.193.222, stratum 2, offset 0.768636, delay 0.02747
 8 Jan 12:58:58 ntpdate[3138]: step time server 81.173.193.222 offset 0.768636 sec
server 81.173.193.222, stratum 2, offset 0.771571, delay 0.02805
 8 Jan 12:59:58 ntpdate[3140]: step time server 81.173.193.222 offset 0.771571 sec
-----


afaik, we can definable see the difference on how the "clock ticking" behaves. right? I mean: using HEAD the offset increases around 0.140207 per minute; with RELEASE/12.1.0 it increases 0.003008



% bc
16.447002 - 16.306795
.140207

0.726101 - 0.723093
.003008
Comment 107 Cy Schubert freebsd_committer 2020-01-08 13:43:57 UTC
Agreed, this is a timecounter issue, not an NTP issue. NTP will only adjust the clock up to 500 PPM, after which it stops. 500 PPM is the limit it will update a faulty clock.

Use kern.timecounter.hardware="HPET".
Comment 108 Vinícius Zavam freebsd_committer 2020-01-08 14:24:52 UTC
(In reply to Cy Schubert from comment #107)

I'm quite sure I did test it before (see Comment 19). will run a test again with 'kern.timecounter.hardware=HPET' and report back. btw, thanks for all time you all are investing here. appreciated
Comment 109 Vinícius Zavam freebsd_committer 2020-01-08 14:26:17 UTC
(In reply to Vinícius Zavam from comment #108)

PS: any thoughts on Why the very same hardware works fine with 12.1-RELEASE? would like to have a better picture of that.
Comment 110 Vinícius Zavam freebsd_committer 2020-01-08 16:19:19 UTC
(In reply to Cy Schubert from comment #107)

1. changed the sysctl;
2. ran `ntpdate ntp1`;
3. `service ntpd onestart`.

after waiting a long time, it kinda looks better indeed. but, as you might see it's still showing a high offset. would it be recommended just to try other timecounters and see what goes - waiting hours to sync? before I waited just a few minutes (<20, maybe)

root@head:~ # uptime
 4:15PM  up  1:41, 2 users, load averages: 0.33, 0.29, 0.26

root@head:~ # ntpq -4 -n -c rl -wp localhost
associd=0 status=0615 leap_none, sync_ntp, 1 event, clock_sync,
version="ntpd 4.2.8p12-a (1)", processor="amd64",
system="FreeBSD/13.0-CURRENT", leap=00, stratum=2, precision=-19,
rootdelay=8.046, rootdisp=20.316, refid=131.188.3.223,
reftime=e1c07e15.87120556  Wed, Jan  8 2020 16:13:09.527,
clock=e1c07e9b.f46a5040  Wed, Jan  8 2020 16:15:23.954, peer=54426, tc=7,
mintc=3, offset=12.405128, frequency=-8.399, sys_jitter=0.840451,
clk_jitter=4.580, clk_wander=0.114, tai=37, leapsec=201701010000,
expire=202006280000

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 0.freebsd.pool.ntp.org
                 .POOL.          16 p    -   64    0    0.000    0.000   0.002
+162.159.200.123 10.48.8.4        3 u  124  128  377    1.899   11.758   1.183
-85.114.128.137  131.188.3.220    2 u   57  128  377    2.247   10.270   1.197
-131.188.3.220   .GPS.            1 u  111  128  377    7.906   10.808   1.330
*131.188.3.223   .PZFs.           1 u    1  128  377    8.046   12.701   2.644
+85.220.190.246  213.172.96.14    2 u   55  128  377    4.577   12.635   1.585
Comment 111 Cy Schubert freebsd_committer 2020-01-08 20:32:11 UTC
(In reply to Vinícius Zavam from comment #110)
What does ntptime say?

What model of Dell is this? what is the CPU model? (You can get that from dmesg.) Do you have other Dell computers of this same model? What are their results?

Use ntpdate or ntpd -G to force a large reset when it boots.
Comment 112 Vinícius Zavam freebsd_committer 2020-01-09 10:01:54 UTC
(In reply to Cy Schubert from comment #111)

I reported the model and hardware specs before:

  * Comment 20
  * Comment 38
  * Comment 43

+ https://www.dell.com/en-us/work/shop/povw/poweredge-r440
+ https://ark.intel.com/content/www/us/en/ark/products/193381/intel-xeon-bronze-3204-processor-8-25m-cache-1-90-ghz.html

the '-g' and '-G' parameters where used already to try working the issue around though:

  * Comment 0

+ ntpd_flags="${ntpd_flags} -4 --logfile=/var/log/ntpd.log --statsdir=/etc/ntp/stats -g -g -g -G"

the machine running stable/12 ran with the different timecounter (HPET) over night, and here are the results:

 - I did ran ntpdate before starting ntpd again.

----------
% grep -i ntp /etc/rc.conf
ntpdate_enable="YES"
ntpd_enable="YES"
ntpd_flags="${ntpd_flags} -4 --logfile=/var/log/ntpd.log --statsdir=/etc/ntp/stats"

% ps afxuww | grep ntp | head -1
root    17401   0.0  0.1 19004 19084  -  Ss   17:09        0:04.86 /usr/sbin/ntpd -4 --logfile=/var/log/ntpd.log --statsdir=/etc/ntp/stats -p /var/db/ntp/ntpd.pid -c /etc/ntp.conf -f /var/db/ntp/ntpd.drift -g

% ls -lash /var/db/ntp/ntpd.pid
4 -rw-r--r--  1 ntpd  ntpd     5B Jan  8 17:10 /var/db/ntp/ntpd.pid

% ntptime 
ntp_gettime() returns code 0 (OK)
  time e1c176f9.27c9cd40  Thu, Jan  9 2020 10:55:05.155, (.155423010),
  maximum error 506023 us, estimated error 1 us, TAI offset 37
ntp_adjtime() returns code 0 (OK)
  modes 0x0 (),
  offset 0.000 us, frequency 439.883 ppm, interval 1 s,
  maximum error 506023 us, estimated error 1 us,
  status 0x2001 (PLL,NANO),
  time constant 6, precision 1.000 us, tolerance 496 ppm,

% ntpq -4 -n -c rl -wp 127.0.0.1
associd=0 status=c018 leap_alarm, sync_unspec, 1 event, no_sys_peer,
version="ntpd 4.2.8p12-a (1)", processor="amd64",
system="FreeBSD/12.1-STABLE", leap=11, stratum=16, precision=-19,
rootdelay=0.000, rootdisp=1.080, refid=STEP, reftime=(no time),
clock=e1c17704.a4498bc5  Thu, Jan  9 2020 10:55:16.641, peer=0, tc=6,
mintc=3, offset=0.000000, frequency=439.883, sys_jitter=0.001907,
clk_jitter=0.002, clk_wander=0.389, tai=37, leapsec=201701010000,
expire=202006280000

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 10.0.5.3        134.130.4.17     2 u   29   64    1    0.276  -68.196  12.686
 10.0.5.4        192.53.103.108   2 u   28   64    1    0.135  -51.048  16.505
 10.0.32.248     129.143.2.23     2 u   30   64    1    1.048  -67.666  12.244
 10.0.32.249     129.143.2.33     2 u   71   64    1    0.912  -50.333   0.002

% date
Thu Jan  9 10:58:23 CET 2020
----------

the very same hardware (also using HPET), running HEAD reports the following:

root@head:~ # ntpq -4 -n -c rl -wp localhost
associd=0 status=0615 leap_none, sync_ntp, 1 event, clock_sync,
version="ntpd 4.2.8p12-a (1)", processor="amd64",
system="FreeBSD/13.0-CURRENT", leap=00, stratum=2, precision=-19,
rootdelay=7.920, rootdisp=9.833, refid=131.188.3.220,
reftime=e1c1766b.77ca8f6e  Thu, Jan  9 2020  9:52:43.467,
clock=e1c177fd.00999011  Thu, Jan  9 2020  9:59:25.002, peer=54425, tc=6,
mintc=3, offset=-0.189879, frequency=3.565, sys_jitter=0.134864,
clk_jitter=0.439, clk_wander=0.010, tai=37, leapsec=201701010000,
expire=202006280000

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 0.freebsd.pool.ntp.org
                 .POOL.          16 p    -   64    0    0.000    0.000   0.002
+162.159.200.123 10.48.8.4        3 u   23   64  377    1.797   -0.190   3.075
+85.114.128.137  131.188.3.220    2 u   22   64  377    2.190    1.941   0.149
*131.188.3.220   .GPS.            1 u    6   64  377    7.920   -0.505   1.675
+131.188.3.223   .PZFs.           1 u   26   64  377    7.956   -0.441   0.079
+85.220.190.246  213.172.96.14    2 u    3   64  377    4.450   -0.312   2.236

root@head:~ # uptime
 9:59AM  up 19:26, 2 users, load averages: 0.25, 0.25, 0.24
Comment 113 Vinícius Zavam freebsd_committer 2020-01-09 10:08:01 UTC
(In reply to Vinícius Zavam from comment #112)

now I do remember it! I reported testing HPET and other timecounters before as a NO FIX for this issue because of that offset we see on the machine running stable/12.

'ACPI-fast' was also used; I did not bother trying 'i8254' or 'dummy' tbh
Comment 114 Vinícius Zavam freebsd_committer 2020-01-09 10:17:27 UTC
(In reply to Vinícius Zavam from comment #112)

forgot to mention that this particular kernel for HEAD is not using 'EARLY_AP_STARTUP' as per Comment 104
Comment 115 Vinícius Zavam freebsd_committer 2020-01-09 10:31:54 UTC
(In reply to Vinícius Zavam from comment #112)

sharing the ntptime taken from HEAD:

root@head:~ # ntptime
ntp_gettime() returns code 0 (OK)
  time e1c17f70.39360d04  Thu, Jan  9 2020 10:31:12.223, (.223481167),
  maximum error 55129 us, estimated error 270 us, TAI offset 37
ntp_adjtime() returns code 0 (OK)
  modes 0x0 (),
  offset -398.303 us, frequency 3.509 ppm, interval 1 s,
  maximum error 55129 us, estimated error 270 us,
  status 0x2001 (PLL,NANO),
  time constant 6, precision 1.000 us, tolerance 496 ppm,
Comment 116 Vinícius Zavam freebsd_committer 2020-01-09 20:18:49 UTC
just as 'FYI', I built bootable USB images of stable/12@r353572 and stable/12@r352547 to try setting up ntpd for testing and both images are having/presenting the very same timecounter issues.

world and kernel (GENERIC) all using their default settings.
Comment 117 Cy Schubert freebsd_committer 2020-01-10 13:47:22 UTC
Just out of interest, how much swap is used?
Comment 118 Vinícius Zavam freebsd_committer 2020-01-10 14:00:12 UTC
(In reply to Cy Schubert from comment #117)

something like nothing; I mean, for an USB "livecd" image (is there any swap to be used?).

* quick note: stable/12@r352480 is OK (it served to branch releng/12.1). I am about to test r352513.
Comment 119 Vinícius Zavam freebsd_committer 2020-01-10 14:07:06 UTC
I think that we now share the opinion that this issue is not directly attached/related to ntpd itself. right? asking just to be sure we are on the same page :)

there's something else causing unexpected conditions that do not play well with NTP running on this particular hardware.

as reported on Comment 118, I should test r352513 in a few. if it works I will walk til we find a problematic revision/commit.
Comment 120 Vinícius Zavam freebsd_committer 2020-01-10 14:44:28 UTC
(In reply to Vinícius Zavam from comment #118)

so, r352513 worked just fine. it must be something between this rev (that works fine) and r352547 (which do not work as expected).
Comment 121 Vinícius Zavam freebsd_committer 2020-01-10 15:27:00 UTC
(In reply to Vinícius Zavam from comment #120)

r352530 is broken (for this particular case I reported over this PR).

so, since r352513 what landed on stable/12 that, IMHO, can be relevant:

 https://svnweb.freebsd.org/base?view=revision&revision=352517
  - https://reviews.freebsd.org/D20793

 https://svnweb.freebsd.org/base?view=revision&revision=352519
  - https://reviews.freebsd.org/D21027

can we invite someone to check if it really makes sense?
Comment 122 Cy Schubert freebsd_committer 2020-01-10 20:52:04 UTC
The only stable/12 commits between r352513 and r352530 are the two you identified.

Can you experiment by removing r352519 and later r352517? Not both at the same time, we need to identify which of the two might be the problem.

I also want to look at r352865. It removes the mlockall() for ntp. mlockall() may circumvent your problem.

I'm still puzzled why only this one machine has the issue. How many physical cores does this machine have? How many NUMA domains does it report?
Comment 123 Vinícius Zavam freebsd_committer 2020-01-13 10:37:59 UTC
Created attachment 210689 [details]
svnlite_diff_stable_12__r352513_r352517.patch
Comment 124 Vinícius Zavam freebsd_committer 2020-01-13 10:38:21 UTC
Created attachment 210690 [details]
svnlite_diff_stable_12__r352513_r352519.patch
Comment 125 Vinícius Zavam freebsd_committer 2020-01-13 10:38:38 UTC
Created attachment 210691 [details]
svnlite_diff_stable_12__r352517_r352519.patch
Comment 126 Vinícius Zavam freebsd_committer 2020-01-13 10:49:49 UTC
(In reply to Cy Schubert from comment #122)

well, it looks like the puzzle is a bit tricky indeed :)

  * r352513 works;
  * r352517 works;
  * r352519 *WITHOUT* r352517 works fine [1];
  * r352519 and further rev do not work.

[1] checked r352513 out, and patched with attachment 210691 [details] (https://bz-attachments.freebsd.org/attachment.cgi?id=210691) *ONLY*.

how did I generated the diff (with security/ca_root_nss installed)?

 # svnlite diff https://svn.freebsd.org/base/stable/12@revA https://svn.freebsd.org/base/stable/12@revB > /tmp/patch_name

how did I apply the patch to r352513?

 # svnlite co https://svn.freebsd.org/base/stable/12@r352513 /usr/src
 # cd /usr/src
 # svnlite patch /tmp/svnlite_diff_stable_12__r352517_r352519.patch
Comment 127 Vinícius Zavam freebsd_committer 2020-01-13 11:40:20 UTC
(In reply to Cy Schubert from comment #122)

one physical socket, one numa domain (numa-domain 0).
Comment 128 Cy Schubert freebsd_committer 2020-01-13 13:13:21 UTC
I've added alc@ to this PR to get his take on this.

Did you also try to remove the mlockall(), r352865, patch to ntpd? r352865 may make ntpd more sensitive to r352517 and r352519.
Comment 129 Vinícius Zavam freebsd_committer 2020-01-13 13:53:36 UTC
(In reply to Cy Schubert from comment #128)

I ran ntpd for a while with `rlimit memlock 32` on its config and it did not help

...but, I used stable/12@r354893 which also has an extra MFC patch related to pmap changes (https://svnweb.freebsd.org/base?view=revision&revision=352541)
Comment 130 Vinícius Zavam freebsd_committer 2020-01-13 14:22:59 UTC
(In reply to Vinícius Zavam from comment #129)

booted back the memstick with r352519 on it and used `rlimit memlock 32` on the ntp.conf: NOT OK.
Comment 131 Cy Schubert freebsd_committer 2020-01-13 15:12:17 UTC
rlimit memlock 32 is not the same as reverting r352865. It occurs after the config file is read while r352865 occurs before. Please try it without r352865.
Comment 132 Vinícius Zavam freebsd_committer 2020-01-13 15:20:15 UTC
would it be fine if I revert it (contrib/ntp/ntpd/ntpd.c) using the very last rev available under stable/12?
Comment 133 Vinícius Zavam freebsd_committer 2020-01-13 15:27:02 UTC
(In reply to Vinícius Zavam from comment #132)

sorry, I meant: revert these guys to a rev previous to r352865:

 . contrib/ntp/ntpd/ntpd.c
 . usr.sbin/ntp/config.h
 . usr.sbin/ntp/ntpd/ntp.conf
Comment 134 Vinícius Zavam freebsd_committer 2020-01-13 15:50:07 UTC
Created attachment 210708 [details]
DIFF_base_stable_12_r356668__without__r352865.patch

patch created using svnlite. reverted to previous rev (see diff).
Comment 135 Cy Schubert freebsd_committer 2020-01-13 20:10:00 UTC
(In reply to Vinícius Zavam from comment #134)
Yes, this would test the sensitivity of the mlockall() removal at ntpd startup. If this resolves the issue, the mlockall() removal from ntpd is incompatible with r352517.

However I cannot see why this would be the case.
Comment 136 Vinícius Zavam freebsd_committer 2020-01-14 09:50:54 UTC
(In reply to Cy Schubert from comment #135)

it did NOT work; I used r356668 *without* r352865 :'(

btw, all memstick images I did test are available here on my side still. if you want that I upload them to any shared server you can send me credentials and I can do it.

the machine is now running the 'r356668 *without* r352865' image (using TSC as timecounter). in a few I should write back its stats/drifts/offsets.
Comment 137 Vinícius Zavam freebsd_committer 2020-01-14 12:41:29 UTC
(In reply to Vinícius Zavam from comment #136)

root@freebsd-builder2:~ # svnlite info /usr/src/
Path: /usr/src
Working Copy Root Path: /usr/src
URL: svn://svn.freebsd.lab.localdomain/base/stable/12
Relative URL: ^/stable/12
Repository Root: svn://svn.freebsd.lab.localdomain/base
Repository UUID: ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f
Revision: 356668
Node Kind: directory
Schedule: normal
Last Changed Author: sbruno
Last Changed Rev: 356665
Last Changed Date: 2020-01-12 20:19:00 +0000 (Sun, 12 Jan 2020)

root@freebsd-builder2:~ # svnlite status /usr/src/
M       /usr/src/contrib/ntp/ntpd/ntpd.c
M       /usr/src/usr.sbin/ntp/config.h
M       /usr/src/usr.sbin/ntp/ntpd/ntp.conf
----------

***
*** below, the live usb image;
***


# uptime
12:28PM  up  2:48, 2 users, load averages: 0.17, 0.18, 0.20
----------

# /usr/sbin/ntpd --logfile=/var/log/ntpd.log --statsdir=/var/db/ntp/stats -p /var/db/ntp/ntpd.pid -c /var/etc_ntp.conf -f /var/db/ntp/ntpd.drift -4 -g
----------

# cat /var/etc_ntp.conf
leapfile "/var/db/ntpd.leap-seconds.list"
pool 0.freebsd.pool.ntp.org iburst
restrict 127.0.0.1
restrict ::1
restrict default limited kod nomodify notrap noquery nopeer
restrict source  limited kod nomodify notrap noquery
statistics clockstats cryptostats loopstats peerstats rawstats sysstats
tos minclock 3 maxclock 6
----------

# ntptime 
ntp_gettime() returns code 0 (OK)
  time e1c83294.c1afaf88  Tue, Jan 14 2020 12:29:08.756, (.756587809),
  maximum error 1218537 us, estimated error 0 us, TAI offset 37
ntp_adjtime() returns code 0 (OK)
  modes 0x0 (),
  offset 0.000 us, frequency 500.000 ppm, interval 1 s,
  maximum error 1218537 us, estimated error 0 us,
  status 0x2001 (PLL,NANO),
  time constant 6, precision 1.000 us, tolerance 496 ppm,
----------

# ntpq -n -4 -c rl -wp localhost
associd=0 status=c613 leap_alarm, sync_ntp, 1 event, spike_detect,
version="ntpd 4.2.8p12-a (1)", processor="amd64",
system="FreeBSD/12.1-STABLE", leap=11, stratum=3, precision=-23,
rootdelay=14.764, rootdisp=1196.413, refid=159.69.150.81,
reftime=e1c83255.24754914  Tue, Jan 14 2020 12:28:05.142,
clock=e1c83313.ba52aef1  Tue, Jan 14 2020 12:31:15.727, peer=9746, tc=6,
mintc=3, offset=0.000000, frequency=500.000, sys_jitter=178.672719,
clk_jitter=0.000, clk_wander=0.000, tai=37, leapsec=201701010000,
expire=202006280000

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 0.freebsd.pool.ntp.org
                 .POOL.          16 p    -   64    0    0.000    0.000   0.000
+82.64.45.50     .GPS.            1 u   40   64    3   31.154  812.666 299.335
+136.243.177.133 131.188.3.222    2 u   39   64    3    9.073  809.245 302.204
#213.209.109.44  10.129.40.211    2 u   40   64    3   11.941  634.287 433.084
#5.9.121.21      131.188.3.220    2 u   36   64    3    9.066  1392.44 519.254
#85.214.255.221  124.216.164.14   2 u   20   64    3   18.800  636.070 466.311
#138.201.64.208  131.188.3.222    2 u    1   64    7    9.095  644.750 466.129
+78.46.253.198   237.17.204.95    2 u   65   64    3    9.259  876.089 245.915
*159.69.150.81   131.188.3.220    2 u   64   64    3    7.593  877.861 244.791
+90.187.7.5      .DCFa.           1 u   48   64    3   19.764  1019.84 257.518
+178.63.247.119  17.253.54.123    2 u   49   64    3    9.280  905.753 250.617
+213.172.105.106 213.172.96.14    2 u   16   64    7    4.479  841.065 314.702
+51.75.67.47     129.70.132.36    3 u   21   64    3    9.749  1422.92 525.558
----------

# ls -lash /var/db/ntp/ntpd.drift
4 -rw-r--r--  1 root  ntpd     8B Jan 14 11:25 /var/db/ntp/ntpd.drift
----------

# cat /var/db/ntp/ntpd.drift
500.000
----------

# cat /var/db/ntp/stats/loopstats
58862 37842.264 0.000000000 500.000 0.000000119 0.000000 6
58862 37851.228 0.114694461 500.000 0.040550616 0.000000 6
58862 38151.836 0.000000000 500.000 0.000000119 0.000000 6
58862 38171.808 0.086842965 500.000 0.030703625 0.000000 6
58862 38206.821 0.092197613 500.000 0.028782938 0.000000 6
58862 38545.516 0.000000000 500.000 0.000000119 0.000000 6
58862 38555.534 0.108082686 500.000 0.038213000 0.000000 6
58862 38606.516 0.122227920 500.000 0.036093146 0.000000 6
58862 38942.190 0.000000000 500.000 0.000000119 0.000000 6
58862 38949.215 0.121292319 500.000 0.042883311 0.000000 6
58862 39257.819 0.000000000 500.000 0.000000119 0.000000 6
58862 39559.249 0.000000000 500.000 0.000000119 0.000000 6
58862 40072.803 0.000000000 500.000 0.000000119 0.000000 6
58862 40502.738 0.000000000 500.000 0.000000119 0.000000 6
58862 40958.617 0.000000000 500.000 0.000000119 0.000000 6
58862 41395.446 0.000000000 500.000 0.000000119 0.000000 6
58862 41806.254 0.000000000 500.000 0.000000119 0.000000 6
58862 42284.875 0.000000000 500.000 0.000000119 0.000000 6
58862 42945.170 0.000000000 500.000 0.000000119 0.000000 6
58862 43514.896 0.000000000 500.000 0.000000119 0.000000 6
58862 44118.361 0.000000000 500.000 0.000000119 0.000000 6
58862 44630.151 0.000000000 500.000 0.000000119 0.000000 6
58862 45208.048 0.000000000 500.000 0.000000119 0.000000 6

# wc /var/db/ntp/stats/peerstats
     754    6032   62443 /var/db/ntp/stats/peerstats

# head -50 /var/db/ntp/stats/peerstats
58862 37531.533 213.209.109.44 1014 1.077890359 0.012070961 7.937500180 0.000000119
58862 37531.553 82.64.45.50 1014 1.079504466 0.032085015 7.937500315 0.000000119
58862 37532.530 136.243.177.133 1014 1.080806975 0.009437440 7.937500190 0.000000119
58862 37533.556 213.209.109.44 1014 1.082719493 0.012121513 3.937507770 0.004829134
58862 37533.576 82.64.45.50 1014 1.084433245 0.031886184 3.937507971 0.004928778
58862 37534.557 136.243.177.133 1014 1.085543716 0.009227317 3.937507783 0.004736741
58862 37535.526 5.9.121.21 1014 1.087885609 0.009139170 7.937500143 0.000000119
58862 37535.529 213.209.109.44 1014 1.087242752 0.011981241 1.937515314 0.007345990
58862 37535.549 82.64.45.50 1014 1.088564439 0.032337657 1.937515553 0.007040947
58862 37536.535 136.243.177.133 1014 1.090188088 0.009078699 1.937515329 0.007401874
58862 37537.506 5.9.121.21 1014 1.092506565 0.009234921 3.937507715 0.004620956
58862 37537.509 213.209.109.44 1014 1.091955616 0.011985290 0.937520962 0.010088748
58862 37539.651 213.209.109.44 1014 0.002468651 0.012097263 7.937500180 0.000000119
58862 37541.649 5.9.121.21 1024 0.007802739 0.009111893 7.937500143 0.000000119
58862 37541.652 213.209.109.44 1014 0.007193125 0.011922935 3.937507769 0.004724473
58862 37543.649 5.9.121.21 1024 0.012528231 0.009260437 3.937507715 0.004725492
58862 37543.649 136.243.177.133 1024 0.012420368 0.009333823 7.937500189 0.000000119
58862 37543.652 213.209.109.44 1014 0.011880933 0.011957675 1.937515314 0.007435274
58862 37545.642 5.9.121.21 1024 0.017204421 0.009149525 1.937515251 0.007424903
58862 37545.642 136.243.177.133 1024 0.017034418 0.009224962 3.937507783 0.004614050
58862 37545.645 213.209.109.44 1014 0.016577549 0.011952335 0.937520961 0.010151956
58862 37547.612 5.9.121.21 1024 0.021821418 0.009220313 0.937520894 0.010069812
58862 37547.612 136.243.177.133 1024 0.021800885 0.009191850 1.937515330 0.007440205
58862 37547.615 213.209.109.44 161a 0.021225211 0.011961550 0.437524722 0.012822093
58862 37549.609 5.9.121.21 1424 0.026448697 0.009266346 0.437524654 0.012731059
58862 37549.610 136.243.177.133 1024 0.026397545 0.009227886 0.937520978 0.010069076
58862 37551.640 5.9.121.21 161a 0.031312830 0.009092092 0.187527001 0.015610121
58862 37551.640 136.243.177.133 1424 0.031181775 0.009229372 0.437524740 0.012874665
58862 37553.635 136.243.177.133 1424 0.035877015 0.009100215 0.187527089 0.015590387
58862 37580.644 213.209.109.44 141a 0.098862243 0.012097064 0.187752307 0.087245607
58862 37607.600 5.9.121.21 141a 0.162869339 0.009261819 0.062927082 0.143572624
58862 37607.623 82.64.45.50 1024 0.163679980 0.032040971 7.937500315 0.000000119
58862 37609.612 82.64.45.50 1024 0.168943986 0.031597416 3.937507969 0.005264006
58862 37611.672 82.64.45.50 1024 0.173166858 0.032193042 1.937515550 0.007342802
58862 37613.620 82.64.45.50 1024 0.178066571 0.031590011 0.937521212 0.010233974
58862 37615.634 82.64.45.50 1024 0.182960282 0.031458396 0.437524979 0.013115317
58862 37617.623 82.64.45.50 1024 0.187942028 0.032190200 0.187527337 0.016063752
58862 37645.649 136.243.177.133 131a 0.252399282 0.009334014 0.063192952 0.228421388
58862 37676.592 5.9.121.21 162a 0.325252766 0.009265608 0.000727142 0.289726091
58862 37706.652 213.209.109.44 142a 0.395491959 0.012302270 0.063556569 0.370594487
58862 37706.672 82.64.45.50 141a 0.396911066 0.031591768 0.063171050 0.221270238
58862 37711.618 136.243.177.133 142a 0.407560780 0.009317224 0.000837798 0.359886482
58862 37738.634 85.214.255.221 1014 0.468418039 0.019533679 7.937500266 0.000000119
58862 37740.659 85.214.255.221 1014 0.472941001 0.019237409 3.937507896 0.004522962
58862 37742.659 85.214.255.221 1014 0.477717306 0.019382327 1.937515463 0.007392208
58862 37744.647 5.9.121.21 143a 0.490381900 0.019403816 0.000865851 0.419574702
58862 37744.647 85.214.255.221 1014 0.482478367 0.019473098 0.937521122 0.010186951
58862 37746.651 85.214.255.221 1014 0.487110119 0.019389783 0.437524888 0.012843635
58862 37772.617 82.64.45.50 162a 0.551768500 0.032194044 0.000826974 0.353053955
58862 37774.600 213.209.109.44 132a 0.555208404 0.012542652 0.001034484 0.494262673

# tail -50 /var/db/ntp/stats/peerstats
58862 44633.151 90.187.7.5 1024 0.637537062 0.021736366 7.937500252 0.000000119
58862 44634.165 51.75.67.47 1034 0.645345351 0.009968341 7.937500149 0.000000119
58862 44635.163 159.69.150.81 1034 0.644559836 0.007822520 7.937500148 0.000000119
58862 44635.164 138.201.64.208 1064 0.644749522 0.009094834 7.937500158 0.000000119
58862 44698.130 78.46.253.198 1014 0.761280623 0.009279784 3.937751802 0.123958674
58862 44700.164 78.46.253.198 1014 0.765024059 0.009266539 1.937637518 0.090337817
58862 44701.162 159.69.150.81 1034 0.766751084 0.007680146 1.937751508 0.086442558
58862 44714.164 178.63.247.119 1024 0.791085693 0.009409210 3.937807718 0.151911016
58862 44723.186 82.64.45.50 1014 0.808524528 0.031606453 3.937841718 0.168421440
58862 44724.112 136.243.177.133 1034 0.809245300 0.009073175 3.937849031 0.172072828
58862 44725.186 82.64.45.50 1014 0.812665961 0.031154007 1.937682417 0.122055513
58862 44729.109 5.9.121.21 1074 0.818480408 0.009120925 1.937864001 0.125693219
58862 44741.137 213.172.105.106 1014 0.841064860 0.004479430 0.938086069 0.149301374
58862 44742.120 51.75.67.47 1034 0.845352059 0.009852784 3.937905223 0.200006708
58862 44744.104 51.75.67.47 1034 0.849026435 0.009851743 1.937714010 0.144047709
58862 44760.134 78.46.253.198 1014 0.876089069 0.009258964 0.937962877 0.165857523
58862 44761.162 159.69.150.81 1034 0.877860560 0.007592801 0.938019651 0.163261743
58862 44776.123 178.63.247.119 1024 0.905753150 0.009280153 0.937980271 0.179391025
58862 44804.165 51.75.67.47 1034 0.960172225 0.009828527 0.938000903 0.203840638
58862 44839.115 90.187.7.5 1024 1.019841188 0.019763973 0.438561726 0.257302837
58862 44913.164 5.9.121.21 1074 1.159282585 0.009118737 0.188933250 0.337521041
58862 45039.102 5.9.121.21 1574 1.392443344 0.009065561 0.064147002 0.519254147
58862 45054.134 51.75.67.47 1434 1.422925249 0.009749120 0.064567247 0.525558154
58862 45121.125 51.75.67.47 1434 1.547027949 0.009718224 0.001532345 0.597976490
58862 45127.139 213.172.105.106 141a 1.556211612 0.004437814 0.002580784 0.604643715
58862 45141.130 138.201.64.208 1564 0.752094280 0.009113566 0.005565666 0.493406740
58862 45145.145 78.46.253.198 141a 1.588989862 0.009257890 0.002427393 0.602722418
58862 45169.116 213.209.109.44 1534 1.382539646 0.012003556 0.003461257 0.401974337
58862 45188.129 51.75.67.47 1434 1.671012142 0.009708984 0.001228419 0.600794495
58862 45190.162 85.214.255.221 1514 1.188130962 0.019081293 0.003548610 0.320168745
58862 45207.107 138.201.64.208 1564 0.866947303 0.009163973 0.005393284 0.514416030
58862 45209.105 78.46.253.198 1014 0.764071175 0.009364904 7.937500368 0.000000119
58862 45209.108 213.209.109.44 1044 0.761240965 0.012549930 7.937500184 0.000000119
58862 45210.106 138.201.64.208 1074 0.765904433 0.009310517 7.937500159 0.000000119
58862 45210.116 85.214.255.221 1024 0.765277075 0.019859849 7.937500268 0.000000119
58862 45210.120 90.187.7.5 1014 0.761396771 0.024153187 7.937500271 0.000000119
58862 45210.128 82.64.45.50 1024 0.766820991 0.032057241 7.937500315 0.000000119
58862 45211.066 51.75.67.47 1044 0.770099725 0.009669124 7.937500147 0.000000119
58862 45212.069 213.172.105.106 1014 0.769277463 0.004626638 7.937500571 0.000000119
58862 45212.074 5.9.121.21 1084 0.769345653 0.009210194 7.937500144 0.000000119
58862 45212.074 178.63.247.119 1034 0.769607200 0.009594431 7.937500146 0.000000119
58862 45213.072 136.243.177.133 1044 0.771148616 0.009084254 7.937500187 0.000000119
58862 45213.084 159.69.150.81 1014 0.774089209 0.013293912 7.937500189 0.000000119
58862 45258.047 159.69.150.81 1014 0.854471430 0.007752637 3.937668992 0.080382221
58862 45260.060 159.69.150.81 1014 0.858243844 0.007731180 1.937595893 0.059566071
58862 45274.058 90.187.7.5 1014 0.879748458 0.023653975 3.937740402 0.118351687
58862 45275.104 178.63.247.119 1034 0.886369234 0.009448902 1.937740255 0.082609415
58862 45284.063 213.209.109.44 1044 0.900025164 0.012018858 1.937785338 0.098358518
58862 45284.083 82.64.45.50 1024 0.903789294 0.031609554 1.937781808 0.096963290
58862 45286.085 5.9.121.21 1084 0.906434434 0.009101309 3.937777715 0.137088781


# cat /var/db/ntp/stats/sysstats
58862 41134.563 3600 743 718 731 0 0 0 0 0 0 0
58862 44741.133 3600 694 694 694 0 0 0 0 0 0 0

# wc /var/db/ntp/stats/rawstats
    3162   53902  577044 /var/db/ntp/stats/rawstats

# head -50 /var/db/ntp/stats/rawstats
58862 37528.567 10.0.86.163 82.64.45.50 0.000000000 0.000000000 2551564513.473209355 0.000000000 3 4 3 0 6 -23 0.000000 0.000000 .INIT.
58862 37529.500 10.0.86.163 136.243.177.133 0.000000000 0.000000000 2568341729.883757025 0.000000000 3 4 3 0 6 -23 0.000000 16777216.000000 .INIT.
58862 37530.508 10.0.86.163 213.209.109.44 0.000000000 0.000000000 2585118945.699312538 0.000000000 3 4 3 0 6 -23 0.000000 33554432.000000 .INIT.
58862 37531.521 10.0.86.163 213.209.109.44 0.000000000 0.000000000 2601896161.646427484 0.000000000 3 4 3 0 6 -23 0.000000 50331648.000000 .INIT.
58862 37531.521 10.0.86.163 82.64.45.50 0.000000000 0.000000000 2601896161.040211291 0.000000000 3 4 3 0 6 -23 0.000000 50331648.000000 .INIT.
58862 37531.521 10.0.86.163 5.9.121.21 0.000000000 0.000000000 2601896161.317005904 0.000000000 3 4 3 0 6 -23 0.000000 50331648.000000 .INIT.
58862 37531.533 213.209.109.44 10.0.86.163 3787986331.520591536 3787986332.604517375 3787986332.604539781 3787986331.532684902 0 4 4 2 6 -24 0.000290 0.024048 10.129.40.211
58862 37531.553 82.64.45.50 10.0.86.163 3787986331.520649614 3787986332.616196588 3787986332.616201415 3787986331.552739456 0 4 4 1 6 -25 0.000000 0.000000 .GPS.
58862 37532.520 10.0.86.163 136.243.177.133 0.000000000 0.000000000 2618673377.145006509 0.000000000 3 4 3 0 6 -23 0.000000 67108864.000000 .INIT.
58862 37532.530 136.243.177.133 10.0.86.163 3787986332.520097682 3787986333.605623377 3787986333.605792078 3787986332.529703823 0 4 4 2 6 -23 0.008667 0.021744 131.188.3.222
58862 37533.544 10.0.86.163 213.209.109.44 2618673377.488750672 2601896161.664205464 2635450593.370243641 0.000000000 3 4 3 0 6 -23 0.000000 83886080.000000 .INIT.
58862 37533.544 10.0.86.163 82.64.45.50 2618673377.239757515 2601896161.247398409 2635450593.836476180 0.000000000 3 4 3 0 6 -23 0.000000 83886080.000000 .INIT.
58862 37533.556 213.209.109.44 10.0.86.163 3787986333.544094584 3787986334.632874834 3787986334.632945914 3787986333.556287177 0 4 4 2 6 -24 0.000290 0.024078 10.129.40.211
58862 37533.576 82.64.45.50 10.0.86.163 3787986333.544145813 3787986334.644522149 3787986334.644526978 3787986333.576036826 0 4 4 1 6 -25 0.000000 0.000000 .GPS.
58862 37534.548 10.0.86.163 136.243.177.133 2635450593.551514960 2618673377.463555963 2652227809.803303394 0.000000000 3 4 3 0 6 -23 0.000000 100663296.000000 .INIT.
58862 37534.557 136.243.177.133 10.0.86.163 3787986334.548014033 3787986335.638171407 3787986335.638243568 3787986334.557313510 0 4 4 2 6 -23 0.008667 0.021774 131.188.3.222
58862 37535.517 10.0.86.163 5.9.121.21 0.000000000 0.000000000 2669005025.568073184 0.000000000 3 4 3 0 6 -23 0.000000 117440512.000000 .INIT.
58862 37535.517 10.0.86.163 213.209.109.44 2652227809.311493434 2635450593.136084113 2669005025.115940125 0.000000000 3 4 3 0 6 -23 0.000000 117440512.000000 .INIT.
58862 37535.517 10.0.86.163 82.64.45.50 2652227809.334854105 2635450593.246680711 2669005025.481388182 0.000000000 3 4 3 0 6 -23 0.000000 117440512.000000 .INIT.
58862 37535.526 5.9.121.21 10.0.86.163 3787986335.516577576 3787986336.609032770 3787986336.609081100 3787986335.525765076 0 4 4 2 6 -25 0.008545 0.029251 131.188.3.220
58862 37535.529 213.209.109.44 10.0.86.163 3787986335.516611940 3787986336.609845313 3787986336.609861169 3787986335.528609037 0 4 4 2 6 -24 0.000290 0.024109 10.129.40.211
58862 37535.549 82.64.45.50 10.0.86.163 3787986335.516620426 3787986336.621353694 3787986336.621358517 3787986335.548962906 0 4 4 1 6 -25 0.000000 0.000000 .GPS.
58862 37536.526 10.0.86.163 136.243.177.133 2669005025.206762531 2652227809.141016755 2685782241.348000319 0.000000000 3 4 3 0 6 -23 0.000000 134217728.000000 .INIT.
58862 37536.535 136.243.177.133 10.0.86.163 3787986336.525575062 3787986337.620302500 3787986337.620373700 3787986336.534724962 0 4 4 2 6 -23 0.008667 0.021805 131.188.3.222
58862 37537.497 10.0.86.163 5.9.121.21 2685782241.186491764 2669005025.244302304 2702559457.688707381 0.000000000 3 4 3 0 6 -23 0.000000 150994944.000000 .INIT.
58862 37537.497 10.0.86.163 213.209.109.44 2685782241.569765068 2669005025.003605993 2702559457.660066634 0.000000000 3 4 3 0 6 -23 0.000000 150994944.000000 .INIT.
58862 37537.497 10.0.86.163 82.64.45.50 2685782241.052155591 2669005025.261070761 2702559457.192552715 0.000000000 3 4 3 0 6 -23 0.000000 150994944.000000 .INIT.
58862 37537.506 5.9.121.21 10.0.86.163 3787986337.496586781 3787986338.593710806 3787986338.593800735 3787986337.505911630 0 4 4 2 6 -25 0.008545 0.029282 131.188.3.220
58862 37537.509 213.209.109.44 10.0.86.163 3787986337.496596972 3787986338.594545233 3787986338.594569266 3787986337.508606294 0 4 4 2 6 -24 0.000290 0.024139 10.129.40.211
58862 37539.639 10.0.86.163 213.209.109.44 0.000000000 0.000000000 2736113889.033362784 0.000000000 3 4 3 0 6 -23 0.000000 0.000000 .STEP.
58862 37539.651 213.209.109.44 10.0.86.163 3787986339.638527514 3787986339.647044797 3787986339.647099708 3787986339.650679689 0 4 4 2 6 -24 0.000290 0.024155 10.129.40.211
58862 37541.640 10.0.86.163 5.9.121.21 0.000000000 0.000000000 2769668321.710583963 0.000000000 3 4 3 0 6 -23 0.000000 33554432.000000 .STEP.
58862 37541.640 10.0.86.163 213.209.109.44 2736113889.583307781 2736113889.687279859 2769668321.749722936 0.000000000 3 4 3 0 6 -23 0.000000 33554432.000000 .STEP.
58862 37541.649 5.9.121.21 10.0.86.163 3787986341.639967484 3787986341.652326169 3787986341.652411091 3787986341.649164299 0 4 4 2 6 -25 0.008545 0.029327 131.188.3.220
58862 37541.652 213.209.109.44 10.0.86.163 3787986341.640013560 3787986341.653168153 3787986341.653238435 3787986341.652006778 0 4 4 2 6 -24 0.000290 0.024185 10.129.40.211
58862 37543.640 10.0.86.163 5.9.121.21 2769668321.794571200 2769668321.658709505 2803222753.789960244 0.000000000 3 4 3 0 6 -23 0.000000 67108864.000000 .STEP.
58862 37543.640 10.0.86.163 213.209.109.44 2769668321.326694170 2769668321.546553233 2803222753.487363257 0.000000000 3 4 3 0 6 -23 0.000000 67108864.000000 .STEP.
58862 37543.640 10.0.86.163 136.243.177.133 0.000000000 0.000000000 2803222753.153958776 0.000000000 3 4 3 0 6 -23 0.000000 67108864.000000 .STEP.
58862 37543.649 5.9.121.21 10.0.86.163 3787986343.639972376 3787986343.657130826 3787986343.657199790 3787986343.649301777 0 4 4 2 6 -25 0.008545 0.029358 131.188.3.220
58862 37543.649 136.243.177.133 10.0.86.163 3787986343.640005657 3787986343.657092937 3787986343.657199758 3787986343.649446302 0 4 4 2 6 -23 0.008667 0.021896 131.188.3.222
58862 37543.652 213.209.109.44 10.0.86.163 3787986343.639995783 3787986343.657855553 3787986343.657899995 3787986343.651997900 0 4 4 2 6 -24 0.000290 0.024216 10.129.40.211
58862 37545.633 10.0.86.163 5.9.121.21 2803222753.829074780 2803222753.166568318 2836777185.207337298 0.000000000 3 4 3 0 6 -23 0.000000 100663296.000000 .STEP.
58862 37545.633 10.0.86.163 213.209.109.44 2803222753.320837775 2803222753.595060924 2836777185.041062393 0.000000000 3 4 3 0 6 -23 0.000000 100663296.000000 .STEP.
58862 37545.633 10.0.86.163 136.243.177.133 2803222753.301731030 2803222753.891056219 2836777185.709427931 0.000000000 3 4 3 0 6 -23 0.000000 100663296.000000 .STEP.
58862 37545.642 5.9.121.21 10.0.86.163 3787986345.633027327 3787986345.654806511 3787986345.654857640 3787986345.642227981 0 4 4 2 6 -25 0.008545 0.029388 131.188.3.220
58862 37545.642 136.243.177.133 10.0.86.163 3787986345.633081299 3787986345.654728199 3787986345.654799398 3787986345.642377461 0 4 4 2 6 -23 0.008667 0.021927 131.188.3.222
58862 37545.645 213.209.109.44 10.0.86.163 3787986345.633064451 3787986345.655618167 3787986345.655673794 3787986345.645072413 0 4 4 2 6 -24 0.000290 0.024246 10.129.40.211
58862 37547.603 10.0.86.163 5.9.121.21 2836777185.073252002 2836777185.566610911 2870331617.050434268 0.000000000 3 4 3 3 6 -23 570621952.000000 1996095488.000000 213.209.109.44
58862 37547.603 10.0.86.163 213.209.109.44 2836777185.875928560 2836777185.208849170 2870331617.982593811 0.000000000 3 4 3 3 6 -23 570621952.000000 1996095488.000000 213.209.109.44
58862 37547.603 10.0.86.163 136.243.177.133 2836777185.941141167 2836777185.417380490 2870331617.854221678 0.000000000 3 4 3 3 6 -23 570621952.000000 1996095488.000000 213.209.109.44

...

58862 45075.728 127.0.0.1 127.0.0.1 775168100.438357246 1630351921.160906322 1881156642.396048513 0.000000000 6 2 3 130 0 1 5062.000000 2516647936.000000 118.101.114.115 372: 73736f723d22616d643634222c0d0a73797374656d3d22467265654253442f31
322e312d535441424c45222c206c6561703d332c207374726174756d3d332c20707265636973696f6e3d2d32332c0d0a726f6f7464656c61793d31342e3736342c20726f6f74646973703d313139362e3431332c2072656669643d3135392e36392e3135302e38312c0d0a72656674696d653d307865316
338333235352e32343735343931342c20636c6f636b3d307865316338333331332e62613532616566312c20706565723d393734362c2074633d362c0d0a6d696e74633d332c206f66667365743d302e3030303030302c206672657175656e63793d3530302e3030302c207379735f6a69747465723d3137
382e3637323731392c0d0a636c6b5f6a69747465723d302e3030302c20636c6b5f77616e6465723d302e3030302c207461693d33372c206c6561707365633d3230313730313031303030302c0d0a6578706972653d3230323030363238303030300d0a0000
58862 45075.730 127.0.0.1 127.0.0.1 605295398.101899275 437522726.390946397 336924454.453446278 0.000000000 6 2 3 129 0 2 5062.000000 872415232.000000 38.22.20.52 16: 260d1534260c1434260b1414260a8811
58862 45075.731 127.0.0.1 127.0.0.1 539766830.439017442 1031041647.449709665 1869112178.133750227 0.000000000 6 2 3 162 0 3 170267016.000000 3556835328.000000 115.114.99.97 432: 302e667265656273642e706f6f6c2e6e74702e6f7267222c0d0a647374616
4723d302e302e302e302c20647374706f72743d302c206c6561703d332c207374726174756d3d31362c20707265636973696f6e3d2d32332c0d0a726f6f7464656c61793d302e3030302c20726f6f74646973703d302e3030302c2072656669643d504f4f4c2c0d0a72656674696d653d30783030303030
3030302e30303030303030302c207265633d307830303030303030302e30303030303030302c2072656163683d3078302c0d0a756e72656163683d302c20686d6f64653d332c20706d6f64653d302c2068706f6c6c3d362c2070706f6c6c3d31302c20686561647761793d302c20666c6173683d3078313
630302c0d0a6b657969643d302c206f66667365743d302e3030302c2064656c61793d302e3030302c2064697370657273696f6e3d31363030302e3030302c206a69747465723d302e3030302c0d0a66696c7464656c61793d20302e303020302e303020302e303020302e303020302e303020302e303020
302e303020302e30302c0d0a66696c746f66667365743d20302e303020302e303020302e303020302e3030
58862 45075.731 127.0.0.1 127.0.0.1 540028974.188235175 1711934764.392401481 1030779753.188326903 0.000000000 6 2 3 130 0 3 170267016.000000 1778439169.000000 32.48.46.48 72: 30302e30302031363030302e30302031363030302e30302031363030302e3030
2031363030302e30302031363030302e30302031363030302e30302031363030302e30300d0a0000
58862 45075.735 127.0.0.1 127.0.0.1 775238702.125674260 1885565555.240058090 741552689.454885729 0.000000000 6 2 3 162 0 4 187044884.000000 3556835328.000000 115.114.99.97 432: 6164723d31302e302e38362e3136332c20647374706f72743d3132332c0d0a
6c6561703d302c207374726174756d3d312c20707265636973696f6e3d2d32352c20726f6f7464656c61793d302e3030302c20726f6f74646973703d302e3030302c0d0a72656669643d4750532c2072656674696d653d307865316338333265632e30303030303030302c207265633d307865316338333
265622e32623639623734622c0d0a72656163683d3078332c20756e72656163683d302c20686d6f64653d332c20706d6f64653d342c2068706f6c6c3d362c2070706f6c6c3d362c20686561647761793d3436382c0d0a666c6173683d3078302c206b657969643d302c206f66667365743d3831322e3636
362c2064656c61793d33312e3135342c2064697370657273696f6e3d36362e3239322c0d0a6a69747465723d3239392e3333352c20786c656176653d302e3032302c0d0a66696c7464656c61793d2033322e35302033312e33312033312e37362033312e34362033312e31352033312e36312033312e373
620302e30302c0d0a66696c746f66667365743d20313338352e383620313135332e313820313033382e35
58862 45075.736 127.0.0.1 127.0.0.1 540489262.180544866 908079669.192111027 774905904.051455509 0.000000000 6 2 3 130 0 4 187044884.000000 1543558145.000000 48.32.57.50 56: 0a66696c74646973703d20302e303020312e383920322e383220332e373520342e
363520342e363820362e30352031363030302e30300d0a
58862 45075.740 127.0.0.1 127.0.0.1 825111348.192111445 539767603.439017442 1031041647.172656190 0.000000000 6 2 3 162 0 5 203830292.000000 3556835328.000000 115.114.99.97 432: 206473746164723d31302e302e38362e3136332c20647374706f72743d3132
332c0d0a6c6561703d302c207374726174756d3d322c20707265636973696f6e3d2d32332c20726f6f7464656c61793d382e3735392c20726f6f74646973703d32332e3830342c0d0a72656669643d3133312e3138382e332e3232322c2072656674696d653d307865316338333164362e6564613330353
7312c0d0a7265633d307865316338333265632e31656665336530392c2072656163683d3078332c20756e72656163683d302c20686d6f64653d332c20706d6f64653d342c0d0a68706f6c6c3d362c2070706f6c6c3d362c20686561647761793d3436392c20666c6173683d3078302c206b657969643d30
2c206f66667365743d3830392e3234352c0d0a64656c61793d392e3037332c2064697370657273696f6e3d36372e3232322c206a69747465723d3330322e3230342c20786c656176653d302e3032352c0d0a66696c7464656c61793d20392e333420392e323620392e333220392e323020392e323020392
e303720392e323020302e30302c0d0a66696c746f66667365743d20313338362e393620313135332e3639
58862 45075.740 127.0.0.1 127.0.0.1 540225582.180468632 941634864.196017813 859185205.215594185 0.000000000 6 2 3 130 0 5 203830292.000000 1660998657.000000 32.49.48.51 64: 20302e30302c0d0a66696c74646973703d20302e303020312e383920322e383220
332e373520342e363520342e363820362e30382031363030302e30300d0a00
58862 45075.746 127.0.0.1 127.0.0.1 825112880.203830313 1931488308.435308662 826111090.125674438 0.000000000 6 2 3 162 0 6 220607509.000000 3556835328.000000 115.114.99.97 432: 6473746164723d31302e302e38362e3136332c20647374706f72743d313233
2c0d0a6c6561703d302c207374726174756d3d322c20707265636973696f6e3d2d32342c20726f6f7464656c61793d302e3330352c20726f6f74646973703d382e3230392c0d0a72656669643d31302e3132392e34302e3231312c2072656674696d653d307865316338333264392e62336631623638312
c0d0a7265633d307865316338333265622e32363239363962362c2072656163683d3078332c20756e72656163683d302c20686d6f64653d332c20706d6f64653d342c0d0a68706f6c6c3d362c2070706f6c6c3d362c20686561647761793d3436382c20666c6173683d3078302c206b657969643d302c20
6f66667365743d3633342e3238372c0d0a64656c61793d31312e3934312c2064697370657273696f6e3d36362e3237302c206a69747465723d3433332e3038342c20786c656176653d302e3034372c0d0a66696c7464656c61793d2031322e30302031322e30312031322e33302031322e32392031322e3
2342031322e32392031312e393420302e30302c0d0a66696c746f66667365743d20313338322e35342031
58862 45075.746 127.0.0.1 127.0.0.1 958410801.203830313 808984633.188311471 875575328.125812005 0.000000000 6 2 3 130 0 6 220607509.000000 1761661953.000000 49.52.56.46 72: 3633342e323920302e30302c0d0a66696c74646973703d20302e303020312e3839
20322e383220332e373520342e363520342e363820362e30362031363030302e30300d0a000000
58862 45075.752 127.0.0.1 127.0.0.1 841888050.449709665 1869636466.192343977 539767602.380683147 0.000000000 6 2 3 162 0 7 237401109.000000 3556835328.000000 115.114.99.97 432: 64723d31302e302e38362e3136332c20647374706f72743d3132332c206c65
61703d302c0d0a7374726174756d3d322c20707265636973696f6e3d2d32352c20726f6f7464656c61793d382e3439392c20726f6f74646973703d34372e3037332c0d0a72656669643d3133312e3138382e332e3232302c2072656674696d653d307865316338326166312e30353566666364622c0d0a7
265633d307865316338333265662e31613261656165642c2072656163683d3078332c20756e72656163683d312c20686d6f64653d332c20706d6f64653d342c0d0a68706f6c6c3d362c2070706f6c6c3d362c20686561647761793d3437322c20666c6173683d3078302c206b657969643d302c206f6666
7365743d313339322e3434332c0d0a64656c61793d392e3036362c2064697370657273696f6e3d36342e3134372c206a69747465723d3531392e3235342c20786c656176653d302e3032372c0d0a66696c7464656c61793d20392e303720392e313220392e323220392e323120392e313220392e3232203
92e323020302e30302c0d0a66696c746f66667365743d20313339322e343420313135392e323820313034
58862 45075.752 127.0.0.1 127.0.0.1 775434552.219241631 925774897.203950894 909585968.188204773 0.000000000 6 2 3 130 0 7 237401109.000000 1593889793.000000 52.46.53.53 60: 302c0d0a66696c74646973703d20302e303020312e383920322e383220332e3735
20342e363520342e363820362e30392031363030302e30300d0a00
58862 45075.758 127.0.0.1 127.0.0.1 892481076.196078194 1931488305.435308662 826111090.125674438 0.000000000 6 2 3 162 0 8 254153749.000000 3556835328.000000 115.114.99.97 432: 6473746164723d31302e302e38362e3136332c20647374706f72743d313233
2c0d0a6c6561703d302c207374726174756d3d322c20707265636973696f6e3d2d32332c20726f6f7464656c61793d31312e3936332c20726f6f74646973703d34322e3438302c0d0a72656669643d3132342e3231362e3136342e31342c2072656674696d653d307865316338326432382e30326136633
166342c0d0a7265633d307865316338333266662e32313563613630392c2072656163683d3078332c20756e72656163683d312c20686d6f64653d332c20706d6f64653d342c0d0a68706f6c6c3d362c2070706f6c6c3d362c20686561647761793d3438382c20666c6173683d3078302c206b657969643d
302c206f66667365743d3633362e3037302c0d0a64656c61793d31382e3830302c2064697370657273696f6e3d36372e3030322c206a69747465723d3436362e3331312c20786c656176653d302e3032322c0d0a66696c7464656c61793d2031392e35372031392e30382031392e34332031392e3535203
1392e35352031392e35322031382e383020302e30302c0d0a66696c746f66667365743d20313432312e34
58862 45075.758 127.0.0.1 127.0.0.1 858665783.207902921 808857144.215640552 540292398.180468811 0.000000000 6 2 3 130 0 8 254153749.000000 1811993601.000000 49.32.49.49 72: 3837203633362e303720302e30302c0d0a66696c74646973703d20302e30302031
2e383920322e383220332e373520342e363520342e363820362e33362031363030302e30300d0a
58862 45075.764 127.0.0.1 127.0.0.1 908996912.188265693 1931488312.435308662 826111090.125674438 0.000000000 6 2 3 162 0 9 270951445.000000 3556835328.000000 115.114.99.97 432: 6473746164723d31302e302e38362e3136332c20647374706f72743d313233
2c0d0a6c6561703d302c207374726174756d3d322c20707265636973696f6e3d2d32342c20726f6f7464656c61793d382e3433382c20726f6f74646973703d32372e3534322c0d0a72656669643d3133312e3138382e332e3232322c2072656674696d653d307865316338333030352e636465323636396
52c0d0a7265633d307865316338333331322e32303339626131362c2072656163683d3078372c20756e72656163683d312c20686d6f64653d332c20706d6f64653d342c0d0a68706f6c6c3d362c2070706f6c6c3d362c20686561647761793d3530302c20666c6173683d3078302c206b657969643d302c
206f66667365743d3634342e3735302c0d0a64656c61793d392e3039352c2064697370657273696f6e3d352e3630342c206a69747465723d3436362e3132392c20786c656176653d302e3032342c0d0a66696c7464656c61793d20392e323120392e313820392e323820392e323220392e313620392e313
920392e313120392e30392c0d0a66696c746f66667365743d20313435372e353520313332392e39302031
58862 45075.764 127.0.0.1 127.0.0.1 909647926.207903755 892679968.125842582 775042359.211429190 0.000000000 6 2 3 130 0 9 270951445.000000 1593889793.000000 48.57.54.46 60: 34342e37352c0d0a66696c74646973703d20302e303020312e303420322e393320
332e383620342e373920352e363920352e373220362e35390d0a00
58862 45075.770 127.0.0.1 127.0.0.1 859124270.219622682 1920147500.447012924 842087796.391115916 0.000000000 6 2 3 162 0 10 287709716.000000 3556835328.000000 115.114.99.97 432: 73746164723d31302e302e38362e3136332c20647374706f72743d3132332
c0d0a6c6561703d302c207374726174756d3d322c20707265636973696f6e3d2d32312c20726f6f7464656c61793d31312e3631322c20726f6f74646973703d33332e3930352c0d0a72656669643d3233372e31372e3230342e39352c2072656674696d653d307865316338326638372e64306538623236
382c0d0a7265633d307865316338333264322e32396632306330632c2072656163683d3078332c20756e72656163683d302c20686d6f64653d332c20706d6f64653d342c0d0a68706f6c6c3d362c2070706f6c6c3d362c20686561647761793d3434332c20666c6173683d3078302c206b657969643d302
c206f66667365743d3837362e3038392c0d0a64656c61793d392e3235392c2064697370657273696f6e3d36352e3431392c206a69747465723d3234352e3931352c20786c656176653d302e3034392c0d0a66696c7464656c61793d20392e323620392e323820392e333820392e323620392e323720392e
323820392e333120302e30302c0d0a66696c746f66667365743d20313333392e313220313130352e373520
58862 45075.770 127.0.0.1 127.0.0.1 909582393.196047676 825636640.125857483 775369526.187991273 0.000000000 6 2 3 130 0 10 287709716.000000 1627444225.000000 57.57.48.46 64: 2e30302c0d0a66696c74646973703d20302e303020312e383920322e383220332
e373520342e363520342e363820352e36392031363030302e30300d0a000000

...

# tail -50 /var/db/ntp/stats/rawstats
58862 45285.096 10.0.86.163 136.243.177.133 3811821793.283310350 3811821793.975198510 3845376225.487924343 0.000000000 3 4 3 0 6 -23 0.000000 1258291200.000000 .STEP.
58862 45285.105 136.243.177.133 10.0.86.163 3787994085.096144228 3787994086.005257523 3787994086.005309717 3787994085.105291032 0 4 4 2 6 -23 0.008759 0.027527 131.188.3.222
58862 45286.076 10.0.86.163 5.9.121.21 2620639457.526025226 2620639457.045422558 3862153441.643134896 0.000000000 3 4 3 0 6 -23 0.000000 1275068416.000000 .STEP.
58862 45286.085 5.9.121.21 10.0.86.163 3787994086.076135912 3787994086.987121001 3787994086.987170904 3787994086.085287125 0 4 4 2 6 -25 0.008514 0.023178 131.188.3.220
58862 45288.091 10.0.86.163 5.9.121.21 3862153441.493098735 3862153441.521008794 3895707873.525394743 0.000000000 3 4 3 0 6 -23 0.000000 1308622848.000000 .STEP.
58862 45288.100 5.9.121.21 10.0.86.163 3787994088.090904267 3787994089.005591680 3787994089.005628622 3787994088.100062891 0 4 4 2 6 -25 0.008514 0.023209 131.188.3.220
58862 45300.040 10.0.86.163 213.172.105.106 2620639457.965499451 2620639457.860956196 4097034465.518361213 0.000000000 3 4 3 0 6 -23 0.000000 1509949440.000000 .STEP.
58862 45300.045 213.172.105.106 10.0.86.163 3787994100.040324421 3787994100.974746887 3787994100.974803586 3787994100.044992071 0 4 4 2 6 -20 0.001648 0.000809 213.172.96.14
58862 45301.036 10.0.86.163 51.75.67.47 2603862241.343136122 2603862241.407911066 4113811681.265170278 0.000000000 3 4 3 0 6 -23 0.000000 1526726656.000000 .STEP.
58862 45301.046 51.75.67.47 10.0.86.163 3787994101.035947934 3787994101.977476639 3787994101.977538857 3787994101.045554730 0 4 4 3 6 -25 0.009369 0.002960 129.70.132.36
58862 45302.064 10.0.86.163 213.172.105.106 4097034465.327065287 4097034465.697654965 4130588897.175893966 0.000000000 3 4 3 0 6 -23 0.000000 1543503872.000000 .STEP.
58862 45302.064 10.0.86.163 85.214.255.221 2587085025.097306840 2587085025.802653975 4130588897.243551496 0.000000000 3 4 3 0 6 -23 0.000000 1543503872.000000 .STEP.
58862 45302.072 213.172.105.106 10.0.86.163 3787994102.064011048 3787994103.002219000 3787994103.002296173 3787994102.071534146 0 4 4 2 6 -20 0.001648 0.000809 213.172.96.14
58862 45302.083 85.214.255.221 10.0.86.163 3787994102.064031198 3787994103.008567078 3787994103.008678016 3787994102.083063499 0 4 4 2 6 -23 0.011963 0.046188 124.216.164.14
58862 45303.096 10.0.86.163 51.75.67.47 4113811681.558536528 4113811681.548731389 4147366113.119134789 0.000000000 3 4 3 0 6 -23 0.000000 1560281088.000000 .STEP.
58862 45303.108 51.75.67.47 10.0.86.163 3787994103.096168465 3787994104.041492098 3787994104.041567690 3787994103.107718816 0 4 4 3 6 -25 0.009369 0.002960 129.70.132.36
58862 45304.096 10.0.86.163 85.214.255.221 4147366113.955935956 4130588897.264255707 4164143329.456872707 0.000000000 3 4 3 0 6 -23 0.000000 1577058304.000000 .STEP.
58862 45304.115 85.214.255.221 10.0.86.163 3787994104.096145001 3787994105.044396789 3787994105.044502776 3787994104.115178756 0 4 4 2 6 -23 0.011963 0.046219 124.216.164.14
58862 45314.072 10.0.86.163 138.201.64.208 2587085025.738188020 2587085025.905258304 37013729.149878089 0.000000000 3 4 3 0 6 -23 0.000000 1744830464.000000 .STEP.
58862 45314.081 138.201.64.208 10.0.86.163 3787994114.071920285 3787994115.035003144 3787994115.035035224 3787994114.081076858 0 4 4 2 6 -24 0.008438 0.031143 131.188.3.222
58862 45316.039 10.0.86.163 138.201.64.208 53790945.519805433 37013729.954891269 70568161.138580444 0.000000000 3 4 3 0 6 -23 0.000000 1778384896.000000 .STEP.
58862 45316.048 138.201.64.208 10.0.86.163 3787994116.039100297 3787994117.005769807 3787994117.005820531 3787994116.048254631 0 4 4 2 6 -24 0.008438 0.031174 131.188.3.222
58862 45319.058 10.0.86.163 78.46.253.198 3409168609.405931707 3409168609.357987707 120899809.477460626 0.000000000 3 4 3 0 6 -23 0.000000 1828716544.000000 .STEP.
58862 45319.067 78.46.253.198 10.0.86.163 3787994119.058047919 3787994120.030293321 3787994120.030319871 3787994119.067352633 0 4 4 2 6 -21 0.011612 0.038544 237.17.204.95
58862 45320.054 10.0.86.163 159.69.150.81 3425945825.747040982 3425945825.229344968 137677025.874599579 0.000000000 3 4 3 3 6 -23 1493499904.000000 3136356608.000000 78.46.253.198
58862 45320.062 159.69.150.81 10.0.86.163 3787994120.053755156 3787994121.026963968 3787994121.027123505 3787994120.061710944 0 4 4 2 6 -24 0.007172 0.028091 131.188.3.220
58862 45335.051 10.0.86.163 178.63.247.119 3677604065.895129859 3677604065.932350522 389335265.835038427 0.000000000 3 4 3 3 6 -23 1493499904.000000 3388014848.000000 78.46.253.198
58862 45335.061 178.63.247.119 10.0.86.163 3787994135.051098217 3787994136.053191144 3787994136.053201539 3787994135.060564965 0 4 4 2 6 -25 0.005371 0.027817 17.253.54.123
58862 45336.054 10.0.86.163 90.187.7.5 3694381281.523389100 3694381281.316005058 406112481.822644296 0.000000000 3 4 3 3 6 -23 1493499904.000000 3404792064.000000 78.46.253.198
58862 45336.077 90.187.7.5 10.0.86.163 3787994136.053979446 3787994137.060615431 3787994137.060630847 3787994136.077218155 0 4 4 1 6 -24 0.000000 0.002502 .DCFa.
58862 45344.054 10.0.86.163 213.209.109.44 3828599009.043319998 3828599009.948717419 540330209.896908346 0.000000000 3 4 3 3 6 -23 1493499904.000000 3539009792.000000 78.46.253.198
58862 45344.054 10.0.86.163 82.64.45.50 3828599009.677390991 3828599009.630694096 540330209.003369096 0.000000000 3 4 3 3 6 -23 1493499904.000000 3539009792.000000 78.46.253.198
58862 45344.066 213.209.109.44 10.0.86.163 3787994144.053857559 3787994145.070982009 3787994145.071021710 3787994144.066056797 0 4 4 2 6 -24 0.000351 0.006836 10.129.9.96
58862 45344.086 82.64.45.50 10.0.86.163 3787994144.053907156 3787994145.084690120 3787994145.084694947 3787994144.086118647 0 4 4 1 6 -25 0.000000 0.000000 .GPS.
58862 45345.042 10.0.86.163 136.243.177.133 3862153441.273351372 3845376225.388106591 557107425.584468605 0.000000000 3 4 3 3 6 -23 1493499904.000000 3539009792.000000 78.46.253.198
58862 45345.051 136.243.177.133 10.0.86.163 3787994145.041940664 3787994146.062204633 3787994146.062280472 3787994145.051393929 0 4 4 2 6 -23 0.008759 0.028427 131.188.3.222
58862 45348.082 10.0.86.163 5.9.121.21 3912485089.608893395 3895707873.733285731 607439073.385771279 0.000000000 3 4 3 3 6 -23 1493499904.000000 3589341440.000000 78.46.253.198
58862 45348.091 5.9.121.21 10.0.86.163 3787994148.081676566 3787994149.107454605 3787994149.107533267 3787994148.090983552 0 4 4 2 6 -25 0.008514 0.024124 131.188.3.220
58862 45362.057 10.0.86.163 213.172.105.106 4147366113.384698272 4130588897.816639904 842320097.147255960 0.000000000 3 4 3 3 6 -23 1493499904.000000 3824222464.000000 78.46.253.198
58862 45362.061 213.172.105.106 10.0.86.163 3787994162.056910643 3787994163.106706988 3787994163.106734188 3787994162.061432005 0 4 4 2 6 -20 0.001831 0.000244 213.172.96.14
58862 45363.056 10.0.86.163 51.75.67.47 4164143329.121805432 4147366113.845544046 859097313.130925003 0.000000000 3 4 3 3 6 -23 1493499904.000000 3840999680.000000 78.46.253.198
58862 45363.066 51.75.67.47 10.0.86.163 3787994163.055870302 3787994164.112263317 3787994164.112327436 3787994163.065631574 0 4 4 3 6 -25 0.009369 0.002991 129.70.132.36
58862 45364.096 10.0.86.163 85.214.255.221 4180920545.681768658 4164143329.872474439 875874529.252496546 0.000000000 3 4 3 3 6 -23 1493499904.000000 3857776896.000000 78.46.253.198
58862 45364.116 85.214.255.221 10.0.86.163 3787994164.096155360 3787994165.155928792 3787994165.156026089 3787994164.115799563 0 4 4 2 6 -23 0.011963 0.047119 124.216.164.14
58862 45376.079 10.0.86.163 138.201.64.208 87345377.306464970 70568161.364904049 1077201121.168582921 0.000000000 3 4 3 3 6 -23 1493499904.000000 4059103488.000000 78.46.253.198
58862 45376.089 138.201.64.208 10.0.86.163 3787994176.079103957 3787994177.157180019 3787994177.157290125 3787994176.088556430 0 4 4 2 6 -24 0.008438 0.032074 131.188.3.222
58862 45381.096 10.0.86.163 78.46.253.198 137677025.019710721 120899809.671954993 1161087201.594095057 0.000000000 3 4 3 3 6 -23 1493499904.000000 4142989568.000000 78.46.253.198
58862 45381.108 78.46.253.198 10.0.86.163 3787994181.096146977 3787994182.185581401 3787994182.185823299 3787994181.108031710 0 4 4 2 6 -21 0.011612 0.039474 237.17.204.95
58862 45382.052 10.0.86.163 159.69.150.81 154454241.904555382 137677025.833157304 1177864417.635473433 0.000000000 3 4 3 3 6 -23 1493499904.000000 4159766784.000000 78.46.253.198
58862 45382.060 159.69.150.81 10.0.86.163 3787994182.052302279 3787994183.140304250 3787994183.140388829 3787994182.060109532 0 4 4 2 6 -24 0.007172 0.029022 131.188.3.220
Comment 138 Cy Schubert freebsd_committer 2020-01-14 12:58:11 UTC
(In reply to Vinícius Zavam from comment #137)
This is excessive and should have been in an attachment.
Comment 139 Cy Schubert freebsd_committer 2020-01-14 13:10:32 UTC
(In reply to Vinícius Zavam from comment #136)
Rather than post full binary images of your memsticks, svn rev number and svn diff from the root of the source tree used to build the images is preferred.
Comment 140 Vinícius Zavam freebsd_committer 2020-01-14 13:26:56 UTC
(In reply to Cy Schubert from comment #139)

yeah, sorry for the big paste before. my bad ;\

regarding the diffs and revs we are all good; we have that here already :)

as I used 'generic' and all default settings to create the USB images, it's pretty easy to reproduce.
Comment 141 Cy Schubert freebsd_committer 2020-01-14 13:35:36 UTC
Given that you tried and had the issue under 13-CURRENT on the Dell R440 and that I run 13-CURRENT on four machines without the issue here, and that you don't have the issue on non-Dell R440, Dell R440 is required to reproduce this problem. Correct?
Comment 142 Vinícius Zavam freebsd_committer 2020-01-15 09:27:16 UTC
(In reply to Cy Schubert from comment #141)

you are correct; maybe other machines using same board and CPU can also have this issue but I do not same such a similar hardware to test it.

dmesg and more information are also available here on this PR. should you guys need any extra detail, just need to ask.

tyvm!
Comment 143 Cy Schubert freebsd_committer 2020-03-13 20:26:55 UTC
The discussion at https://lists.freebsd.org/pipermail/freebsd-current/2020-March/075472.html describes this problem. See kib@'s summary of the problem at the end of the thread : https://lists.freebsd.org/pipermail/freebsd-current/2020-March/075489.html.