This looks like a regression because it worked prior to an upgrade to FreeBSD 14 and works *everywhere* else (other *BSD's, all Linux variants, etc). kenj@vm10:~/src/pcp/qa$ python3 -c 'from time import tzset' Traceback (most recent call last): File "<string>", line 1, in <module> ImportError: cannot import name 'tzset' from 'time' (unknown location) My python is the bog standard package from FreeBSD 14, namely kenj@vm10:~/src/pcp/qa$ pkg info python3 python3-3_3 Name : python3 Version : 3_3 Installed on : Wed Jan 10 15:40:56 2024 AEDT Origin : lang/python3 Architecture : FreeBSD:14:* Prefix : /usr/local Categories : python lang Licenses : Maintainer : python@FreeBSD.org WWW : https://www.python.org/ Comment : Meta-port for the Python interpreter 3.x Annotations : build_timestamp: 2023-12-23T03:14:23+0000 built_by : poudriere-git-3.4.0 port_checkout_unclean: no port_git_hash : 2b5c95373 ports_top_checkout_unclean: no ports_top_git_hash: 9d5f5f8e2 repo_type : binary repository : FreeBSD
Python here seems to be the victim, not the cause. localtime() is not working correctly for FreeBSD 14.0 and is different to FreeBSD 13.2 and Linux variants, and this is causing this code fragment in lang/python39's configure.ac to fail ... # check tzset(3) exists and works like we expect it to ... time_t groundhogday = 1044144000; /* GMT-based */ ... putenv("TZ=AEST-10AEDT-11,M10.5.0,M3.5.0"); tzset(); if (localtime(&groundhogday)->tm_hour != 11) exit(1); and so HAVE_WORKING_TZSET is NOT defined and so the tzset method is omitted from the time module. I've extracted the code fragment from configure.ac and FreeBSD 14.0 returns the incorrect value 10 for tm_hour.
if this isn't a bug in python, then maybe this bug should be moved to base. or rather, we should maybe create the reproducer as base bug and link them
Could the local timezone mentioned here have changed?
Created attachment 247802 [details] Reproducer C program compile and run ... should emit OK for success, assorted Botch messages for failure
I'm no bugzilla wizard, nor a FreeBSD bug workflow expert, so please do what you need to ... the root cause appears to be in libc's localtime(). But please note the Python problem is a build-time issue, not a run time issue, so once localtime() is fixed, Python will need to be re-built and re-released. The issue has nothing to do with the system's timezone, the reproducer snippet I included originally explicitly sets TZ in the environment, calls tzset() and localtime() returns the wrong result. I've attached a small C program that reproduces the problem, and here is my output: kenj@vm06:~$ cc groundhog.c kenj@vm06:~$ a.out OK I am: FreeBSD 13.2-RELEASE-p4 GENERIC 13.2-RELEASE-p4 kenj@vm10:~$ cc groundhog.c kenj@vm10:~$ a.out Botch hour=10 not 11 Botch isdst=0 not 1 I am: FreeBSD 14.0-RELEASE-p3 #0: Mon Dec 11 04:54:25 UTC 2023 root@amd64-builder.daemonology.net:/usr/obj/usr/src/i386.i386/sys/GENERIC 14.0-RELEASE-p3
My only concern is that the test depends on a timezone whose definitions have changed in all the TZ cleanup that happened and thus the problem is now in the test. It looks like it is an hour off and isdst also mismastches. That may be a fine answer. I don’t completely grok the end of the TZ string so maybe that is overriding the normal TZ definition. It would be good to know if that's the issue or not. We've imported both new TZ definitions and updated our TZ parsing code and maybe this indicates a regression in the code.
The test sets time_t to 1044144000 which is GMT Sun Feb 2 00:00:00 2003. The M10.5.0,M3.5.0 part of the test TZ means - daylight saving starts on the last (5) Sunday (0) of Oct (10) - daylight saving ends on the last (5) Sunday (0) of Mar (3) So our test time should be in the daylight saving period. In addition to failing on FreeBSD 14.0 and passing on FreeBSD 13.2, the same test passes on Ubuntu 22.04, Ubuntu 16.04, Ubuntu 14.04, Arch Linux, OpenBSD 7.3, RHEL 6.10, Fedora 39, Debian 12.4, Fedora 38, Ubuntu 18.04, ... so it seems unlikely that FreeBSD 14.0 alone is "correct".
(In reply to Ken McDonell from comment #7) So a couple of things on my 14.0-RC2 machine and a 15.0-current (jan 10th): % env TZ=AEST-10AEDT-11,M10.5.0,M3.5.0 date -r 1044144000 Sun Feb 2 11:00:00 AEDT 2003 So that looks right. That eliminates at least the parsing code from being the problem. A similar test on a linux box gives: % env TZ=AEST-10AEDT-11,M10.5.0,M3.5.0 date --date='@1044144000' Sun Feb 2 11:00:00 AEDT 2003 But then I run your reproducer test against 14.0-RC2 and I get: % ./foo OK I am: FreeBSD 14.0-RC2 #0 releng/14.0-n265367-7bdb67cb1cfe: Thu Oct 26 17:30:50 MDT 2023 imp@zooty:/usr/home/imp/obj/usr/home/imp/git/releng-14.0/amd64.amd64/sys/GENERIC 14.0-RC2 and % /foo OK I am: FreeBSD 15.0-CURRENT #58 main-n267479-18469e00dc45-dirty: Wed Jan 10 16:31:01 MST 2024 imp@rebo:/usr/home/imp/obj/usr/home/imp/git/head/amd64.amd64/sys/GENERIC 15.0-CURRENT So now I'm confused. I hate 'works for me' bugs... I wonder how your system may differ from mine? Granted, none of these are exactly 14.0, but I'd expect if it worked in 14.0-RC2 but not in 14.0 release p3, I'd expect 15.0 also to be bad as well... Especially since I don't see any 14.0 time changes since the release (the last one being April 26th, 2023, well predating my RC2 system. So I'd like to find out why your system is different, but I'm unsure the best way to do that. The only thing I can think of is architecture. What architecture are you running this on? All my tests are amd64. Did I see an i386 in there somewhere?
ding ding ding We have a winner % ./foo OK I am: FreeBSD 15.0-CURRENT #58 main-n267479-18469e00dc45-dirty: Wed Jan 10 16:31:01 MST 2024 imp@rebo:/usr/home/imp/obj/usr/home/imp/git/head/amd64.amd64/sys/GENERIC 15.0-CURRENT % ./foo-32 Botch hour=10 not 11 Botch isdst=0 not 1 I am: FreeBSD 15.0-CURRENT #58 main-n267479-18469e00dc45-dirty: Wed Jan 10 16:31:01 MST 2024 imp@rebo:/usr/home/imp/obj/usr/home/imp/git/head/amd64.amd64/sys/GENERIC 15.0-CURRENT % file ./foo-32 ./foo-32: ELF 32-bit LSB executable, Intel 80386, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 15.0 (1500008), FreeBSD-style, with debug_info, not stripped % file ./foo ./foo: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 15.0 (1500008), FreeBSD-style, with debug_info, not stripped And this suggests it may be i386 specific. On my aarch64 machine: % ./foo OK I am: FreeBSD 15.0-CURRENT #16 main-n267501-62e8ccc3a489: Wed Jan 10 17:11:20 MST 2024 imp@brazos:/usr/home/imp/obj/usr/home/imp/git/freebsd/arm64.aarch64/sys/GENERIC 15.0-CURRENT % ./foo-32 OK I am: FreeBSD 15.0-CURRENT #16 main-n267501-62e8ccc3a489: Wed Jan 10 17:11:20 MST 2024 imp@brazos:/usr/home/imp/obj/usr/home/imp/git/freebsd/arm64.aarch64/sys/GENERIC 15.0-CURRENT % file foo foo-32 foo: ELF 64-bit LSB executable, ARM aarch64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 15.0 (1500008), FreeBSD-style, with debug_info, not stripped foo-32: ELF 32-bit LSB executable, ARM, EABI5 version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, FreeBSD-style, for FreeBSD 15.0 (1500008), with debug_info, not stripped So (a) this is clearly a base system bug and (b) only appears to affect i386 (though I've not tested on powerpc since I have no powerpc gear running FreeBSD).
Execellent work Warner ... indeed my failing 14.0 system is i686 and my passing 13.2 system is amd64. Thanks.
This seems to be a bug in tzcode, as recently updated from tzcode2010m to tzcode2023c. The bug is still present upstream, checked most recent tzcode2024a and github. More specifically, the tzparse() function in localtime.c uses the increment_overflow_time() function, assuming that the resulting variable ("janfirst" variable) is not changed when the overflow is detected: https://github.com/eggert/tz/blob/e48c5b532a26e094db24a7de44c33786860c7ae1/localtime.c#L1187 This assumption, however, is not true if increment_overflow_time() is implemented using C23 ckd_add() or __builtin_add_overflow(): the overflowed result is stored to the variable instead. This manifests itself on systems with 32-bit time_t, where the loop in question reaches the overflow. The most trival workaround I can see would be to compile localtime.c with HAVE_STDCKDINT_H defined to 0, so that C implementation of increment_overflow_time() is used instead of __builtin_add_overflow(). The C implementation does not change the resulting variable on overflow, so the assumption of the code holds true. Here is a simple test with the most recent tzcode sources, tzcode2024a, on FreeBSD 13.2-RELEASE-p8 i386: $ make date cc -O2 -pipe -c date.c -o date.o printf '%s\n' >tzdir.h.out '#ifndef TZDEFAULT' '# define TZDEFAULT "/etc/localtime" /* default zone */' '#endif' '#ifndef TZDIR' '# define TZDIR "/usr/share/zoneinfo" /* TZif directory */' '#endif' mv tzdir.h.out tzdir.h cc -O2 -pipe -c localtime.c -o localtime.o cc -O2 -pipe -c strftime.c -o strftime.o cc -O2 -pipe -c asctime.c -o asctime.o cc -o date -O2 -pipe date.o localtime.o strftime.o asctime.o $ TZ=AEST-10AEDT-11,M10.5.0,M3.5.0 ./date -r 1044144000 Sun Feb 2 10:00:00 AEST 2003 As can be seen from "10:00:00 AEST", the result is broken. The same code, recompiled with HAVE_STDCKDINT_H set to 0: $ touch localtime.c $ make date CFLAGS=-DHAVE_STDCKDINT_H=0 cc -DHAVE_STDCKDINT_H=0 -c localtime.c -o localtime.o cc -o date -DHAVE_STDCKDINT_H=0 date.o localtime.o strftime.o asctime.o $ TZ=AEST-10AEDT-11,M10.5.0,M3.5.0 ./date -r 1044144000 Sun Feb 2 11:00:00 AEDT 2003 Now the result is correct, "11:00:00 AEDT". It would be great if someone can take a look and implement a workaround.
Fixed upstream: https://github.com/eggert/tz/commit/9fc11a27163ee5e59214d49711142c5750040e98
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=1365bb722af1365baee6ea1e3d44917533908d53 commit 1365bb722af1365baee6ea1e3d44917533908d53 Author: Dag-Erling Smørgrav <des@FreeBSD.org> AuthorDate: 2024-02-18 09:48:08 +0000 Commit: Dag-Erling Smørgrav <des@FreeBSD.org> CommitDate: 2024-02-18 09:51:49 +0000 tzcode: Fix overflow handling in TZ parser. Obtained from: upstream 9fc11a27 MFC after: 1 week PR: 276281 contrib/tzcode/localtime.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
^Triage: cleanup leftovers and set possible mfc flags (please close if N/A).
Can someone please confirm that this patch fixes the issue?
The patch works fine in my tests when compiled separately (with tzcode2024a): $ make date cc -O2 -pipe -c date.c -o date.o printf '%s\n' >tzdir.h.out '#ifndef TZDEFAULT' '# define TZDEFAULT "/etc/localtime" /* default zone */' '#endif' '#ifndef TZDIR' '# define TZDIR "/usr/share/zoneinfo" /* TZif directory */' '#endif' mv tzdir.h.out tzdir.h cc -O2 -pipe -c localtime.c -o localtime.o cc -O2 -pipe -c strftime.c -o strftime.o cc -O2 -pipe -c asctime.c -o asctime.o cc -o date -O2 -pipe date.o localtime.o strftime.o asctime.o $ TZ=AEST-10AEDT-11,M10.5.0,M3.5.0 ./date -r 1044144000 Sun Feb 2 11:00:00 AEDT 2003 Note "11:00:00 AEDT", which is the correct result. I'll test with full OS build to see if it fixes Python build shortly, though I don't expect any surprises here.
A commit in branch stable/14 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=245844372d7e4dd7c633816b67c0da59be75b812 commit 245844372d7e4dd7c633816b67c0da59be75b812 Author: Dag-Erling Smørgrav <des@FreeBSD.org> AuthorDate: 2024-02-18 09:48:08 +0000 Commit: Dag-Erling Smørgrav <des@FreeBSD.org> CommitDate: 2024-02-21 20:33:23 +0000 tzcode: Fix overflow handling in TZ parser. Obtained from: upstream 9fc11a27 MFC after: 1 week PR: 276281 (cherry picked from commit 1365bb722af1365baee6ea1e3d44917533908d53) contrib/tzcode/localtime.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=beb958dca02abf79a2172e702c2d24bbccde60fa commit beb958dca02abf79a2172e702c2d24bbccde60fa Author: Dag-Erling Smørgrav <des@FreeBSD.org> AuthorDate: 2024-02-18 09:48:08 +0000 Commit: Dag-Erling Smørgrav <des@FreeBSD.org> CommitDate: 2024-02-21 20:59:26 +0000 tzcode: Fix overflow handling in TZ parser. Obtained from: upstream 9fc11a27 MFC after: 1 week PR: 276281 Approved by: re (cperciva) (cherry picked from commit 1365bb722af1365baee6ea1e3d44917533908d53) contrib/tzcode/localtime.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
For the record, I've checked Python build on a patched system (FreeBSD 13.2-RELEASE i386 with the patch), and it happily detects tzset() and provides time.tzset(). Thanks for fixing this!
A commit in branch releng/13.3 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=e99479f817c4714d396de1d403daa6f9bc3a32bb commit e99479f817c4714d396de1d403daa6f9bc3a32bb Author: Dag-Erling Smørgrav <des@FreeBSD.org> AuthorDate: 2024-02-18 09:48:08 +0000 Commit: Dag-Erling Smørgrav <des@FreeBSD.org> CommitDate: 2024-02-23 11:41:25 +0000 tzcode: Fix overflow handling in TZ parser. Obtained from: upstream 9fc11a27 MFC after: 1 week PR: 276281 Approved by: re (cperciva) (cherry picked from commit 1365bb722af1365baee6ea1e3d44917533908d53) (cherry picked from commit beb958dca02abf79a2172e702c2d24bbccde60fa) contrib/tzcode/localtime.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
I've upgraded my FreeBSD 14.0 system but the bug remains (both in libc and python). I don't know which package delivers the libc fix, but the Python I have after upgrading is python39-3.9.18_2. Can someone please tell me in *which* version of FreeBSD these fixes will be delivered?
13.4, 14.1 and 15.0 will have the fix. However, in the case of Python, the bug manifests at build time, not at run time, so the official Python packages will remain broken until the package builders are upgraded.