Created attachment 195334 [details] patch(1) to disallow escaping random bits Hi, Please do an exp-run with the attached patch to disallow escaping of arbitrary ordinary characters in regex(3). This makes a smoother progression into the future with GNU extensions where some ordinary characters are blessed with special meaning. Only characters that actually have special meaning when escaped are allowed. This patch addresses some POSIX-declared implementation defined behavior.
Buildworld fails: /poudriere/jails/headi386PR229925/usr/src/lib/libc/regex/regcomp.c:458:9: error: use of undeclared identifier 'BACKSL' if (c&BACKSL && may_escape(p, wc) != 0) ^
Created attachment 195381 [details] patch(1) to disallow escaping random bits, v2 ... I'm sorry for wasting your time. =( This was an older version of the patch, taken from the wrong branch, that had some rebase mishaps thrown on top of that and I failed to test it again. Uploaded is the newer version- re-tested to make sure that it applies and does what I expect.
Exp-run can't start: make: RE substitution error: trailing backslash (\) make: Unclosed substitution for (/ missing) make: "/usr/ports/Mk/bsd.port.mk" line 1464: Undefined variable "${:Upython:C/\:.*//}" make: RE substitution error: trailing backslash (\) make: Unclosed substitution for (/ missing) make: "/usr/ports/Mk/bsd.port.mk" line 1466: Undefined variable "${:Upython:C/^[^\:]*(\:|\$)//:" make: RE substitution error: trailing backslash (\) make: Unclosed substitution for (/ missing) make: "/usr/ports/Mk/bsd.port.mk" line 1470: Cannot open /usr/ports/Mk/Uses/.mk
make: RE substitution error: trailing backslash (\) make: Unclosed substitution for (/ missing) make: "/usr/ports/Mk/bsd.port.mk" line 1464: Undefined variable "${:Upython:C/\:.*//}" make: RE substitution error: trailing backslash (\) make: Unclosed substitution for (/ missing) make: "/usr/ports/Mk/bsd.port.mk" line 1466: Undefined variable "${:Upython:C/^[^\:]*(\:|\$)//:" make: RE substitution error: trailing backslash (\) make: Unclosed substitution for (/ missing) make: "/usr/ports/Mk/bsd.port.mk" line 1464: Undefined variable "${:Ucompiler\:features:C/\:.*//}" make: RE substitution error: trailing backslash (\) make: Unclosed substitution for (/ missing) make: "/usr/ports/Mk/bsd.port.mk" line 1464: Undefined variable "${:Uobjc:C/\:.*//}" make: RE substitution error: trailing backslash (\) make: Unclosed substitution for (/ missing) make: "/usr/ports/Mk/bsd.port.mk" line 1470: Cannot open /usr/ports/Mk/Uses/.mk make: RE substitution error: trailing backslash (\) make: Unclosed substitution for (/ missing) make: "/usr/ports/Mk/bsd.port.mk" line 1470: Cannot open /usr/ports/Mk/Uses/.mk make: RE substitution error: trailing backslash (\) make: Unclosed substitution for (/ missing) make: "/usr/ports/Mk/bsd.port.mk" line 1470: Cannot open /usr/ports/Mk/Uses/.mk make: Fatal errors encountered -- cannot continuemake: Unknown modifier ':' make: Unknown modifier ':' make: Unknown modifier ':' make: RE substitution error: trailing backslash (\) make: Unclosed substitution for (/ missing) make: "/usr/ports/Mk/bsd.port.mk" line 1464: Undefined variable "${:Upython:C/\:.*//}" make: RE substitution error: trailing backslash (\) make: Unclosed substitution for (/ missing) make: "/usr/ports/Mk/bsd.port.mk" line 1466: Undefined variable "${:Upython:C/^[^\:]*(\:|\$)//:" make: RE substitution error: trailing backslash (\) make: Unclosed substitution for (/ missing) make: "/usr/ports/Mk/bsd.port.mk" line 1464: Undefined variable "${:Ucompiler\:features:C/\:.*//}" make: RE substitution error: trailing backslash (\) make: Unclosed substitution for (/ missing) make: "/usr/ports/Mk/bsd.port.mk" line 1464: Undefined variable "${:Uobjc:C/\:.*//}" make: RE substitution error: trailing backslash (\) make: Unclosed substitution for (/ missing) make: "/usr/ports/Mk/bsd.port.mk" line 1470: Cannot open /usr/ports/Mk/Uses/.mk make: RE substitution error: trailing backslash (\) make: Unclosed substitution for (/ missing) make: "/usr/ports/Mk/bsd.port.mk" line 1470: Cannot open /usr/ports/Mk/Uses/.mk make: RE substitution error: trailing backslash (\) make: Unclosed substitution for (/ missing) make: "/usr/ports/Mk/bsd.port.mk" line 1470: Cannot open /usr/ports/Mk/Uses/.mk make: Unknown modifier ':' make: Unknown modifier ':' make: Unknown modifier ':' make: Unknown modifier ':' make: Unknown modifier ':'
So this is going to be totally craptastic... Perfect. Perhaps we should start off with disallowing A-Za-z from being arbitrarily escaped, because this is apparently going to be incredibly painful. On the positive side, these failures are legit- either the escaping is too heavy and make would have done the right thing, or make needed it but is still passing the escape down through to regcomp.
According to make(1): Each modifier begins with a colon, which may be escaped with a backslash (`\').
(In reply to Antoine Brodin from comment #6) Right, and it turns out escaping its use in the :C modifier is not necessary, e.g. [1] removes the gratuitous escaping from these USES bits that caused that particular breakage, and in theory works. I'm doing some grep-work to see if I can pare down the amount of :C usage that I need to whisk through to figure out what all breaks in the ports framework, because it doesn't look as rampant as I had thought from some initial pokes. Given that I don't quite have the resources to build the entire ports tree, do you happen to have a smaller set of ports that you would recommend trying to build on my own as a basic lemon test before asking you to try again? Perhaps a small set that flexes quite a bit of ports functionality. [1] https://people.freebsd.org/~kevans/escaped-colon.diff
Created attachment 195653 [details] svn(1) diff against base, version 3 Take 3... loosened up the restrictions a lot to just restrict [A-Za-z0-9'`] so the fallout won't be quite as catastrophic while still doing what I need it to do. Having received no input on a minimal set of ports to try, I did a poudriere test with: games/gzdoom devel/git devel/arduino18 shells/zsh and hit approximately 300 ports. I will upload my initial ports patch following this one.
Created attachment 195654 [details] svn(1) diff against ports Patch a single escaped ` in www/w3m.
New failures on amd64: + {"origin"=>"databases/freetds", "phase"=>"patch", "errortype"=>"???"} + {"origin"=>"databases/xapian-core", "phase"=>"build", "errortype"=>"bad_C++_code"} + {"origin"=>"databases/xapian-core10", "phase"=>"build", "errortype"=>"clang"} + {"origin"=>"databases/xapian-core12", "phase"=>"build", "errortype"=>"bad_C++_code"} + {"origin"=>"devel/devtodo", "phase"=>"patch", "errortype"=>"???"} + {"origin"=>"devel/erlang-exmpp", "phase"=>"build", "errortype"=>"???"} + {"origin"=>"devel/llvm33", "phase"=>"stage", "errortype"=>"perl"} + {"origin"=>"devel/synfig", "phase"=>"build", "errortype"=>"bad_C++_code"} + {"origin"=>"devel/tig", "phase"=>"build", "errortype"=>"bad_C++_code"} + {"origin"=>"ftp/rexx-curl", "phase"=>"patch", "errortype"=>"???"} + {"origin"=>"german/hunspell", "phase"=>"build", "errortype"=>"???"} + {"origin"=>"java/classpath", "phase"=>"build", "errortype"=>"???"} + {"origin"=>"lang/gcc6", "phase"=>"build", "errortype"=>"assumes_gcc"} + {"origin"=>"mail/libmilter", "phase"=>"configure", "errortype"=>"???"} + {"origin"=>"mail/sendmail", "phase"=>"configure", "errortype"=>"???"} + {"origin"=>"mail/squirrelmail-sasql-plugin", "flavor"=>"php70", "phase"=>"patch", "errortype"=>"???"} + {"origin"=>"mail/squirrelmail-sasql-plugin", "flavor"=>"php71", "phase"=>"patch", "errortype"=>"???"} + {"origin"=>"mail/squirrelmail-sasql-plugin", "flavor"=>"php72", "phase"=>"patch", "errortype"=>"???"} + {"origin"=>"mail/squirrelmail-sasql-plugin", "phase"=>"patch", "errortype"=>"???"} + {"origin"=>"mail/thunderbird", "phase"=>"build", "errortype"=>"clang"} + {"origin"=>"math/octave", "phase"=>"build", "errortype"=>"???"} + {"origin"=>"multimedia/libkate", "phase"=>"patch", "errortype"=>"???"} + {"origin"=>"net-mgmt/nagiosql", "phase"=>"patch", "errortype"=>"???"} + {"origin"=>"net/ipxe", "phase"=>"build", "errortype"=>"missing_header"} + {"origin"=>"net/miniupnpc", "phase"=>"stage", "errortype"=>"???"} + {"origin"=>"security/clamav-unofficial-sigs", "phase"=>"build", "errortype"=>"???"} + {"origin"=>"security/fsh", "phase"=>"build", "errortype"=>"???"} + {"origin"=>"security/openvpn", "phase"=>"build", "errortype"=>"???"} + {"origin"=>"security/openvpn-devel", "phase"=>"build", "errortype"=>"???"} + {"origin"=>"security/openvpn-mbedtls", "phase"=>"build", "errortype"=>"???"} + {"origin"=>"sysutils/file", "phase"=>"build", "errortype"=>"???"} + {"origin"=>"www/firefox", "phase"=>"build", "errortype"=>"clang"} + {"origin"=>"www/firefox-esr", "phase"=>"build", "errortype"=>"clang"} + {"origin"=>"www/foswiki", "phase"=>"patch", "errortype"=>"???"} + {"origin"=>"www/p5-Gantry", "phase"=>"patch", "errortype"=>"???"} + {"origin"=>"www/palemoon", "phase"=>"build", "errortype"=>"linker_error"} + {"origin"=>"www/seamonkey", "phase"=>"configure", "errortype"=>"configure_error"} + {"origin"=>"www/waterfox", "phase"=>"configure", "errortype"=>"configure_error"} + {"origin"=>"x11-toolkits/p5-Alien-wxWidgets", "phase"=>"patch", "errortype"=>"???"} + {"origin"=>"x11-wm/ede", "phase"=>"build", "errortype"=>"bad_C++_code"} Hundreds of ports were skipped due to those failures
New failure logs on amd64: http://package18.nyi.freebsd.org/data/headamd64PR229925-default/2018-08-03_22h32m51s/logs/errors/freetds-1.00.94,1.log http://package18.nyi.freebsd.org/data/headamd64PR229925-default/2018-08-03_22h32m51s/logs/errors/xapian-core-1.4.7_1,1.log http://package18.nyi.freebsd.org/data/headamd64PR229925-default/2018-08-03_22h32m51s/logs/errors/xapian-core10-1.0.23_1.log http://package18.nyi.freebsd.org/data/headamd64PR229925-default/2018-08-03_22h32m51s/logs/errors/xapian-core12-1.2.25,1.log http://package18.nyi.freebsd.org/data/headamd64PR229925-default/2018-08-03_22h32m51s/logs/errors/devtodo-0.1.20_2.log http://package18.nyi.freebsd.org/data/headamd64PR229925-default/2018-08-03_22h32m51s/logs/errors/erlang-exmpp-0.9.9_1.log http://package18.nyi.freebsd.org/data/headamd64PR229925-default/2018-08-03_22h32m51s/logs/errors/llvm33-3.3_14.log http://package18.nyi.freebsd.org/data/headamd64PR229925-default/2018-08-03_22h32m51s/logs/errors/synfig-1.2.1_5.log http://package18.nyi.freebsd.org/data/headamd64PR229925-default/2018-08-03_22h32m51s/logs/errors/tig-2.3.0_1.log http://package18.nyi.freebsd.org/data/headamd64PR229925-default/2018-08-03_22h32m51s/logs/errors/rexx-curl-2.0.1.log http://package18.nyi.freebsd.org/data/headamd64PR229925-default/2018-08-03_22h32m51s/logs/errors/de-hunspell-20161207_1.log http://package18.nyi.freebsd.org/data/headamd64PR229925-default/2018-08-03_22h32m51s/logs/errors/classpath-0.99_7.log http://package18.nyi.freebsd.org/data/headamd64PR229925-default/2018-08-03_22h32m51s/logs/errors/gcc6-6.4.0_8.log http://package18.nyi.freebsd.org/data/headamd64PR229925-default/2018-08-03_22h32m51s/logs/errors/libmilter-8.15.2_1.log http://package18.nyi.freebsd.org/data/headamd64PR229925-default/2018-08-03_22h32m51s/logs/errors/sendmail-8.15.2_11.log http://package18.nyi.freebsd.org/data/headamd64PR229925-default/2018-08-03_22h32m51s/logs/errors/squirrelmail-sasql-plugin-php70-3.2.0_3.log http://package18.nyi.freebsd.org/data/headamd64PR229925-default/2018-08-03_22h32m51s/logs/errors/squirrelmail-sasql-plugin-php71-3.2.0_3.log http://package18.nyi.freebsd.org/data/headamd64PR229925-default/2018-08-03_22h32m51s/logs/errors/squirrelmail-sasql-plugin-php72-3.2.0_3.log http://package18.nyi.freebsd.org/data/headamd64PR229925-default/2018-08-03_22h32m51s/logs/errors/squirrelmail-sasql-plugin-php56-3.2.0_3.log http://package18.nyi.freebsd.org/data/headamd64PR229925-default/2018-08-03_22h32m51s/logs/errors/thunderbird-60.0_1.log http://package18.nyi.freebsd.org/data/headamd64PR229925-default/2018-08-03_22h32m51s/logs/errors/octave-4.4.0_3.log http://package18.nyi.freebsd.org/data/headamd64PR229925-default/2018-08-03_22h32m51s/logs/errors/libkate-0.4.1_7.log http://package18.nyi.freebsd.org/data/headamd64PR229925-default/2018-08-03_22h32m51s/logs/errors/nagiosql-3.2.0_2.log http://package18.nyi.freebsd.org/data/headamd64PR229925-default/2018-08-03_22h32m51s/logs/errors/ipxe-20180514_1.log http://package18.nyi.freebsd.org/data/headamd64PR229925-default/2018-08-03_22h32m51s/logs/errors/miniupnpc-2.1_1.log http://package18.nyi.freebsd.org/data/headamd64PR229925-default/2018-08-03_22h32m51s/logs/errors/clamav-unofficial-sigs-5.6.2.log http://package18.nyi.freebsd.org/data/headamd64PR229925-default/2018-08-03_22h32m51s/logs/errors/fsh-1.2_3.log http://package18.nyi.freebsd.org/data/headamd64PR229925-default/2018-08-03_22h32m51s/logs/errors/openvpn-2.4.6_1.log http://package18.nyi.freebsd.org/data/headamd64PR229925-default/2018-08-03_22h32m51s/logs/errors/openvpn-devel-201821_1.log http://package18.nyi.freebsd.org/data/headamd64PR229925-default/2018-08-03_22h32m51s/logs/errors/openvpn-mbedtls-2.4.6_1.log http://package18.nyi.freebsd.org/data/headamd64PR229925-default/2018-08-03_22h32m51s/logs/errors/file-5.32.log http://package18.nyi.freebsd.org/data/headamd64PR229925-default/2018-08-03_22h32m51s/logs/errors/firefox-61.0.1_1,1.log http://package18.nyi.freebsd.org/data/headamd64PR229925-default/2018-08-03_22h32m51s/logs/errors/firefox-esr-60.1.0_4,1.log http://package18.nyi.freebsd.org/data/headamd64PR229925-default/2018-08-03_22h32m51s/logs/errors/foswiki-2.1.6.log http://package18.nyi.freebsd.org/data/headamd64PR229925-default/2018-08-03_22h32m51s/logs/errors/p5-Gantry-3.64_1.log http://package18.nyi.freebsd.org/data/headamd64PR229925-default/2018-08-03_22h32m51s/logs/errors/palemoon-27.9.4.log http://package18.nyi.freebsd.org/data/headamd64PR229925-default/2018-08-03_22h32m51s/logs/errors/seamonkey-2.49.4_10.log http://package18.nyi.freebsd.org/data/headamd64PR229925-default/2018-08-03_22h32m51s/logs/errors/waterfox-56.2.2.log http://package18.nyi.freebsd.org/data/headamd64PR229925-default/2018-08-03_22h32m51s/logs/errors/p5-Alien-wxWidgets-0.69.log http://package18.nyi.freebsd.org/data/headamd64PR229925-default/2018-08-03_22h32m51s/logs/errors/ede-2.1_3.log
For my own record: llvm33 has been retired since this exp-run was ran, so no patch will be required for it.
net/ipxe should hopefully be fine now after ports r485643.
(In reply to Tobias Kortkamp from comment #13) Thanks!
Comment on attachment 195654 [details] svn(1) diff against ports Marking patch against ports as obsolete; bug #233417 created to track.
I will be submitting PRs for the rest of the initial failures soon... for my records, the following have all also been overcome by events: - sysutils/file (update removed offending expression) - www/palemoon (removed from ports) - www/seamonkey (removed from ports) - www/waterfox (removed from ports)
Created attachment 207331 [details] svn(1) diff against the ports tree This is the small collection of ports patches that are still pending approval as of today; can we apply these and do another exp-run to see what else might be broken, or do I need to 'just be patient'? The base patch still applies. Thanks!
On i386, here are the build logs that contain "RE error:" http://package23.nyi.freebsd.org/data/headi386PR229925-default/2019-09-19_13h24m11s/logs/amavisd-new-2.12.0,1.log http://package23.nyi.freebsd.org/data/headi386PR229925-default/2019-09-19_13h24m11s/logs/apcupsd-3.14.14_2.log http://package23.nyi.freebsd.org/data/headi386PR229925-default/2019-09-19_13h24m11s/logs/asterisk-chan_sccp-4.3.2_1.log http://package23.nyi.freebsd.org/data/headi386PR229925-default/2019-09-19_13h24m11s/logs/bareos16-client-16.2.8_3.log http://package23.nyi.freebsd.org/data/headi386PR229925-default/2019-09-19_13h24m11s/logs/bareos16-client-static-16.2.8_3.log http://package23.nyi.freebsd.org/data/headi386PR229925-default/2019-09-19_13h24m11s/logs/bareos16-server-16.2.8_3.log http://package23.nyi.freebsd.org/data/headi386PR229925-default/2019-09-19_13h24m11s/logs/bareos17-client-17.2.7_4.log http://package23.nyi.freebsd.org/data/headi386PR229925-default/2019-09-19_13h24m11s/logs/bareos17-client-static-17.2.7_4.log http://package23.nyi.freebsd.org/data/headi386PR229925-default/2019-09-19_13h24m11s/logs/bareos17-server-17.2.7_4.log http://package23.nyi.freebsd.org/data/headi386PR229925-default/2019-09-19_13h24m11s/logs/cassandra3-3.11.4_1.log http://package23.nyi.freebsd.org/data/headi386PR229925-default/2019-09-19_13h24m11s/logs/cinnamon-control-center-2.4.2_8.log http://package23.nyi.freebsd.org/data/headi386PR229925-default/2019-09-19_13h24m11s/logs/cinnamon-settings-daemon-2.4.3_6.log http://package23.nyi.freebsd.org/data/headi386PR229925-default/2019-09-19_13h24m11s/logs/conky-1.11.5_1.log http://package23.nyi.freebsd.org/data/headi386PR229925-default/2019-09-19_13h24m11s/logs/conky-awesome-1.11.5_1.log http://package23.nyi.freebsd.org/data/headi386PR229925-default/2019-09-19_13h24m11s/logs/couchdb2-2.3.1_2.log http://package23.nyi.freebsd.org/data/headi386PR229925-default/2019-09-19_13h24m11s/logs/docsis-0.9.8.log http://package23.nyi.freebsd.org/data/headi386PR229925-default/2019-09-19_13h24m11s/logs/etl-1.2.1.log http://package23.nyi.freebsd.org/data/headi386PR229925-default/2019-09-19_13h24m11s/logs/evilvte-0.5.2.p1_2.log http://package23.nyi.freebsd.org/data/headi386PR229925-default/2019-09-19_13h24m11s/logs/ftgl-2.1.3.r5_6,1.log http://package23.nyi.freebsd.org/data/headi386PR229925-default/2019-09-19_13h24m11s/logs/gtk2-2.24.32.log http://package23.nyi.freebsd.org/data/headi386PR229925-default/2019-09-19_13h24m11s/logs/gtk3-3.24.10_1.log http://package23.nyi.freebsd.org/data/headi386PR229925-default/2019-09-19_13h24m11s/logs/libappindicator-12.10.0_2.log http://package23.nyi.freebsd.org/data/headi386PR229925-default/2019-09-19_13h24m11s/logs/libgnomecups-0.2.3_8,1.log http://package23.nyi.freebsd.org/data/headi386PR229925-default/2019-09-19_13h24m11s/logs/minipro-0.3_1.log http://package23.nyi.freebsd.org/data/headi386PR229925-default/2019-09-19_13h24m11s/logs/octopus-8.4_2.log http://package23.nyi.freebsd.org/data/headi386PR229925-default/2019-09-19_13h24m11s/logs/openjdk11-11.0.4+11.2_1.log http://package23.nyi.freebsd.org/data/headi386PR229925-default/2019-09-19_13h24m11s/logs/openjdk12-12.0.2+10.2_1.log http://package23.nyi.freebsd.org/data/headi386PR229925-default/2019-09-19_13h24m11s/logs/openjdk13-13+33.1.log http://package23.nyi.freebsd.org/data/headi386PR229925-default/2019-09-19_13h24m11s/logs/p5-NetCDF-1.2.4_6.log http://package23.nyi.freebsd.org/data/headi386PR229925-default/2019-09-19_13h24m11s/logs/pdflib-perl-7.0.5_5.log http://package23.nyi.freebsd.org/data/headi386PR229925-default/2019-09-19_13h24m11s/logs/rpm4-4.14.2.1_1.log http://package23.nyi.freebsd.org/data/headi386PR229925-default/2019-09-19_13h24m11s/logs/rxvt-2.6.4_6.log http://package23.nyi.freebsd.org/data/headi386PR229925-default/2019-09-19_13h24m11s/logs/sphinxsearch-2.2.11_3.log http://package23.nyi.freebsd.org/data/headi386PR229925-default/2019-09-19_13h24m11s/logs/synfig-1.2.1_20.log http://package23.nyi.freebsd.org/data/headi386PR229925-default/2019-09-19_13h24m11s/logs/synfigstudio-1.2.1_11.log http://package23.nyi.freebsd.org/data/headi386PR229925-default/2019-09-19_13h24m11s/logs/tarantool-2.1.2_1.log http://package23.nyi.freebsd.org/data/headi386PR229925-default/2019-09-19_13h24m11s/logs/vifm-0.10.log http://package23.nyi.freebsd.org/data/headi386PR229925-default/2019-09-19_13h24m11s/logs/vlc-3.0.8,4.log http://package23.nyi.freebsd.org/data/headi386PR229925-default/2019-09-19_13h24m11s/logs/zh-celvis-1.3.log http://package23.nyi.freebsd.org/data/headi386PR229925-default/2019-09-19_13h24m11s/logs/zziplib-0.13.69_1.log
On amd64, here are the build logs that contain "RE error:" http://package22.nyi.freebsd.org/data/headamd64PR229925-default/2019-09-19_13h33m48s/logs/amavisd-new-2.12.0,1.log http://package22.nyi.freebsd.org/data/headamd64PR229925-default/2019-09-19_13h33m48s/logs/apcupsd-3.14.14_2.log http://package22.nyi.freebsd.org/data/headamd64PR229925-default/2019-09-19_13h33m48s/logs/asterisk-chan_sccp-4.3.2_1.log http://package22.nyi.freebsd.org/data/headamd64PR229925-default/2019-09-19_13h33m48s/logs/bareos16-client-16.2.8_3.log http://package22.nyi.freebsd.org/data/headamd64PR229925-default/2019-09-19_13h33m48s/logs/bareos16-client-static-16.2.8_3.log http://package22.nyi.freebsd.org/data/headamd64PR229925-default/2019-09-19_13h33m48s/logs/bareos16-server-16.2.8_3.log http://package22.nyi.freebsd.org/data/headamd64PR229925-default/2019-09-19_13h33m48s/logs/bareos17-client-17.2.7_4.log http://package22.nyi.freebsd.org/data/headamd64PR229925-default/2019-09-19_13h33m48s/logs/bareos17-client-static-17.2.7_4.log http://package22.nyi.freebsd.org/data/headamd64PR229925-default/2019-09-19_13h33m48s/logs/bareos17-server-17.2.7_4.log http://package22.nyi.freebsd.org/data/headamd64PR229925-default/2019-09-19_13h33m48s/logs/cassandra3-3.11.4_1.log http://package22.nyi.freebsd.org/data/headamd64PR229925-default/2019-09-19_13h33m48s/logs/cinnamon-control-center-2.4.2_8.log http://package22.nyi.freebsd.org/data/headamd64PR229925-default/2019-09-19_13h33m48s/logs/cinnamon-settings-daemon-2.4.3_6.log http://package22.nyi.freebsd.org/data/headamd64PR229925-default/2019-09-19_13h33m48s/logs/conky-1.11.5_1.log http://package22.nyi.freebsd.org/data/headamd64PR229925-default/2019-09-19_13h33m48s/logs/conky-awesome-1.11.5_1.log http://package22.nyi.freebsd.org/data/headamd64PR229925-default/2019-09-19_13h33m48s/logs/couchdb2-2.3.1_2.log http://package22.nyi.freebsd.org/data/headamd64PR229925-default/2019-09-19_13h33m48s/logs/docsis-0.9.8.log http://package22.nyi.freebsd.org/data/headamd64PR229925-default/2019-09-19_13h33m48s/logs/etl-1.2.1.log http://package22.nyi.freebsd.org/data/headamd64PR229925-default/2019-09-19_13h33m48s/logs/evilvte-0.5.2.p1_2.log http://package22.nyi.freebsd.org/data/headamd64PR229925-default/2019-09-19_13h33m48s/logs/ftgl-2.1.3.r5_6,1.log http://package22.nyi.freebsd.org/data/headamd64PR229925-default/2019-09-19_13h33m48s/logs/gtk2-2.24.32.log http://package22.nyi.freebsd.org/data/headamd64PR229925-default/2019-09-19_13h33m48s/logs/gtk3-3.24.10_1.log http://package22.nyi.freebsd.org/data/headamd64PR229925-default/2019-09-19_13h33m48s/logs/knot2-2.8.3_1.log http://package22.nyi.freebsd.org/data/headamd64PR229925-default/2019-09-19_13h33m48s/logs/knot2-lib-2.8.3_1.log http://package22.nyi.freebsd.org/data/headamd64PR229925-default/2019-09-19_13h33m48s/logs/libappindicator-12.10.0_2.log http://package22.nyi.freebsd.org/data/headamd64PR229925-default/2019-09-19_13h33m48s/logs/libgnomecups-0.2.3_8,1.log http://package22.nyi.freebsd.org/data/headamd64PR229925-default/2019-09-19_13h33m48s/logs/minipro-0.3_1.log http://package22.nyi.freebsd.org/data/headamd64PR229925-default/2019-09-19_13h33m48s/logs/octopus-8.4_2.log http://package22.nyi.freebsd.org/data/headamd64PR229925-default/2019-09-19_13h33m48s/logs/openjdk11-11.0.4+11.2_1.log http://package22.nyi.freebsd.org/data/headamd64PR229925-default/2019-09-19_13h33m48s/logs/openjdk12-12.0.2+10.2_1.log http://package22.nyi.freebsd.org/data/headamd64PR229925-default/2019-09-19_13h33m48s/logs/openjdk13-13+33.1.log http://package22.nyi.freebsd.org/data/headamd64PR229925-default/2019-09-19_13h33m48s/logs/openmpi-1.10.7_7.log http://package22.nyi.freebsd.org/data/headamd64PR229925-default/2019-09-19_13h33m48s/logs/openmpi2-2.1.6_3.log http://package22.nyi.freebsd.org/data/headamd64PR229925-default/2019-09-19_13h33m48s/logs/openmpi3-3.1.4_2.log http://package22.nyi.freebsd.org/data/headamd64PR229925-default/2019-09-19_13h33m48s/logs/p5-NetCDF-1.2.4_6.log http://package22.nyi.freebsd.org/data/headamd64PR229925-default/2019-09-19_13h33m48s/logs/pdflib-perl-7.0.5_5.log http://package22.nyi.freebsd.org/data/headamd64PR229925-default/2019-09-19_13h33m48s/logs/pianod2-301_2.log http://package22.nyi.freebsd.org/data/headamd64PR229925-default/2019-09-19_13h33m48s/logs/rpm4-4.14.2.1_1.log http://package22.nyi.freebsd.org/data/headamd64PR229925-default/2019-09-19_13h33m48s/logs/rxvt-2.6.4_6.log http://package22.nyi.freebsd.org/data/headamd64PR229925-default/2019-09-19_13h33m48s/logs/sphinxsearch-2.2.11_3.log http://package22.nyi.freebsd.org/data/headamd64PR229925-default/2019-09-19_13h33m48s/logs/synfig-1.2.1_20.log http://package22.nyi.freebsd.org/data/headamd64PR229925-default/2019-09-19_13h33m48s/logs/synfigstudio-1.2.1_11.log http://package22.nyi.freebsd.org/data/headamd64PR229925-default/2019-09-19_13h33m48s/logs/tarantool-2.1.2_1.log http://package22.nyi.freebsd.org/data/headamd64PR229925-default/2019-09-19_13h33m48s/logs/vifm-0.10.log http://package22.nyi.freebsd.org/data/headamd64PR229925-default/2019-09-19_13h33m48s/logs/vlc-3.0.8,4.log http://package22.nyi.freebsd.org/data/headamd64PR229925-default/2019-09-19_13h33m48s/logs/zh-celvis-1.3.log http://package22.nyi.freebsd.org/data/headamd64PR229925-default/2019-09-19_13h33m48s/logs/zziplib-0.13.69_1.log
Thanks, Antoine! Given the magnitude, should I continue with 1 PR : Port, or would it be OK to batch up the more trivial patches and seek portmgr approval to just-do-it? Specifically, I propose batching: - Trivial port makefile escape fixes - Changes from sed to gsed, specifically those that can be done with either BINARY_ALIAS or autoconf env vars. Anything involving patches being added or extra replacements I'd prefer passing through makntaoners, but I'd like to think I'm qualified to make some of the lower-impact decisions that I listed as batchable being effectively one of the only regex(3) maintainers
(In reply to Kyle Evans from comment #20) I prefer that you use PR for everything, but you can group them (for instance 1 PR for bareos*, 1 PR for conky*, 1 PR for openjdk*, 1 PR for openmpi*)
According to https://www.freebsd.org/doc/en/books/porters-handbook/makefile-maintainer.html: Blanket approval for most ports applies to fixes like infrastructure changes, or trivial and tested build and runtime fixes. The current list is available in Ports section of the Committer's Guide. And in https://www.freebsd.org/doc/en_US.ISO8859-1/articles/committers-guide/ports.html#ports-qa-misc-blanket-approval it links to: Blanket approval for most ports applies to these types of fixes: Most infrastructure changes to a port (that is, modernizing, but not changing the functionality). For example, the blanket covers converting to new USES macros, enabling verbose builds, and switching to new ports system syntaxes. Trivial and tested build and runtime fixes. There is absolutely no need to involve portmgr, and you probably do not need to involve the affected ports' maintainers. As you are not a ports committer, you do need to seek approval (by a ports committer) for any patch to the ports tree you wish to commit yourself. If you do not wish to commit the patches yourself, either 1 PR/port or 1 PR for everything is up to you, as in both cases, you only need an ok from a ports committer.
Perfect, thanks to both of you.
Note to self: chinese/celvis is a false positive, source literally contains "RE error" as part of its RE implementation, and it happens to pop up in a compiler warning.
I've opened up a review https://reviews.freebsd.org/D22750 for base that will knock out a large chunk of these that are wanting to use sed \t. They're currently silently failing to match, so I don't expect them suddenly working to cause problems -- thus, I'm not requesting an exp-run with this patch specifically.
A commit references this bug: Author: kevans Date: Tue Dec 10 19:16:01 UTC 2019 New revision: 355590 URL: https://svnweb.freebsd.org/changeset/base/355590 Log: sed: process \r, \n, and \t This is both reasonable and a common GNUism that a lot of ported software expects. Universally process \r, \n, and \t into carriage return, newline, and tab respectively. Newline still doesn't function in contexts where it can't (e.g. BRE), but we process it anyways rather than passing UB \n (escaped ordinary) through to the underlying regex engine. Adding a --posix flag to disable these was considered, but sed.1 already declares this version of sed a super-set of POSIX specification and this behavior is the most likely expected when one attempts to use one of these escape sequences in pattern space. This differs from pre-r197362 behavior in that we now honor the three arguably most common escape sequences used with sed(1) and we do so outside of character classes, too. Other escape sequences, like \s and \S, will come later when GNU extensions are added to libregex; sed will likely link against libregex by default, since the GNU extensions tend to be fairly un-intrusive. PR: 229925 Reviewed by: bapt, emaste, pfg Differential Revision: https://reviews.freebsd.org/D22750 Changes: head/usr.bin/sed/compile.c head/usr.bin/sed/tests/regress.multitest.out/8.22 head/usr.bin/sed/tests/sed2_test.sh
A commit references this bug: Author: kevans Date: Sun Jun 7 04:32:39 UTC 2020 New revision: 361884 URL: https://svnweb.freebsd.org/changeset/base/361884 Log: sed: attempt to learn about hex escapes (e.g. \x27) Somewhat predictably, software often wants to use \x27/\x24 among others so that they can decline worrying about ugly escaping, if said escaping is even possible. Right now, this software is using these and getting the wrong results, as we'll interpret those as x27 and x24 respectively. Some examples of this, when an exp-run was ran, were science/octopus and misc/vifm. Go ahead and process these at all times. We allow either one or two digits, and the tests account for both. If extra digits are specified, e.g. \x2727, then the third and fourth digits are interpreted literally as one might expect. PR: 229925 MFC after: 2 weeks Changes: head/usr.bin/sed/compile.c head/usr.bin/sed/tests/sed2_test.sh
A chunk of these are OBE because I taught sed how to \r, \n, \t, and \xNN in the interim. I picked up new devel/llvm* breakage when testing and opened PR 247058 for that. The rest will be addressed by https://reviews.freebsd.org/D25185, then I'll request one last run. I fully intend to power through this time, rather than neglecting it for 8-12 months...
A commit references this bug: Author: kevans Date: Mon Jun 8 04:41:37 UTC 2020 New revision: 538197 URL: https://svnweb.freebsd.org/changeset/ports/538197 Log: Multiple ports: improve regex compliance These changes largely fall into just two categories: 1. Need textproc/gsed for GNU extensions 2. Extraneous escapes that can go away For #1, there's a further subdivision into those that require autoconf magic and those that can get away with BINARY_ALIAS=sed=${LOCALBASE}/bin/gsed. -CURRENT will soon gain GNU extensions, but these will take longer to get to all supported releases; we must switch them to gsed to ensure we're actually properly building them as intended. For #2, I've fixed these as I can and we should upstream these fixes. PORTREVISION is bumped for all of the above, because we will almost certainly build these differently when the replacements actually start working. These were all detected by the below-referenced exp-run [1]. The patch included forbids many ordinary characters from being escaped, since we'll later imbue those with special meanings. This has had the nice side effect of picking up various things that we didn't handle properly, e.g. \t and \r for tab and carriage return. PR: 229925 [1] Approved by: koobs (mentor) Approved by: portmgr (blanket: trivial build fixes) MFH: no (invasive risk) Differential Revision: https://reviews.freebsd.org/D25185 Changes: head/archivers/rpm4/Makefile head/audio/pianod2/Makefile head/databases/cassandra3/Makefile head/databases/couchdb2/Makefile head/databases/tarantool/Makefile head/devel/etl/Makefile head/devel/libappindicator/Makefile head/devel/libappindicator/files/extra-patch-bindings_vala_examples_Makefile.am head/devel/synfig/Makefile head/devel/zziplib/Makefile head/dns/knot2/Makefile head/graphics/ftgl/files/ head/graphics/ftgl/files/patch-configure head/graphics/synfigstudio/Makefile head/java/openjdk11/Makefile head/java/openjdk12/Makefile head/java/openjdk13/Makefile head/math/p5-NetCDF/files/patch-port_master.mk.in head/multimedia/vlc/Makefile head/net-mgmt/docsis/files/patch-doc_generate-settings.sh head/print/pdflib/Makefile head/print/pdflib/files/patch-configure head/security/amavisd-new/Makefile head/sysutils/cinnamon-control-center/Makefile head/sysutils/cinnamon-settings-daemon/Makefile head/sysutils/conky/Makefile head/sysutils/minipro/Makefile head/textproc/sphinxsearch/Makefile head/x11/evilvte/Makefile
Antoine, can you please run this one more time with ports >= r538368 (10 minutes ago) and base >= r361884 (2 days ago), with just the single patch attached applied to base? I di confirm that the patch still applies as I expect. Between sed improvements and the mass switching of ports over to textproc/gsed in the interim, I suspect this should be largely successful.
Some new failures logs on i386: http://pb2.nyi.freebsd.org/data/headi386PR229925-default/2020-06-27_18h26m15s/logs/errors/cassandra4-4.0.a4.log http://pb2.nyi.freebsd.org/data/headi386PR229925-default/2020-06-27_18h26m15s/logs/errors/couchdb3-3.1.0_1.log http://pb2.nyi.freebsd.org/data/headi386PR229925-default/2020-06-27_18h26m15s/logs/errors/openfortivpn-1.14.1.log http://pb2.nyi.freebsd.org/data/headi386PR229925-default/2020-06-27_18h26m15s/logs/errors/xca-2.3.0.log Some build success with sed error log on i386: http://pb2.nyi.freebsd.org/data/headi386PR229925-default/2020-06-25_06h07m06s/logs/openjdk14-14.0.1+7.1.log
Additional new failure log on amd64: http://package18.nyi.freebsd.org/data/headamd64PR229925-default/2020-06-28_05h52m17s/logs/errors/fr-aster-14.4.0.1_3.log
A commit references this bug: Author: kevans Date: Fri Jul 24 17:10:52 UTC 2020 New revision: 543267 URL: https://svnweb.freebsd.org/changeset/ports/543267 Log: Multiple ports: improve regex compliance (part 2) This is again a part of the project to stop extraneous escaping of ordinary characters and redefine some ordinary escapes as special behavior. Most of these ports are pushed over to use textproc/gsed because they want to use GNU extensions. Others are fixed to either escape appropriately (e.g. $$ rather than \$ in Makefiles!) or just remove redundant escapes (e.g. backtick in single quotes doesn't need escaped). PR: 229925 MFH: no (invasive risk) Changes: head/databases/cassandra4/Makefile head/databases/couchdb3/Makefile head/french/aster/files/patch-as_setup.py head/java/openjdk14/Makefile head/security/openfortivpn/Makefile head/security/xca/Makefile
(In reply to commit-hook from comment #33) Hi Antoine, This should address the rest of the failures; are you satisfied with the results of this exp-run, or should we run it again? Thanks!
(In reply to Kyle Evans from comment #34) This should be fine
(In reply to Antoine Brodin from comment #35) Excellent, thank you very much.
A commit references this bug: Author: kevans Date: Wed Jul 29 23:21:57 UTC 2020 New revision: 363679 URL: https://svnweb.freebsd.org/changeset/base/363679 Log: regex(3): Interpret many escaped ordinary characters as EESCAPE In IEEE 1003.1-2008 [1] and earlier revisions, BRE/ERE grammar allows for any character to be escaped, but "ORD_CHAR preceded by an unescaped <backslash> character [gives undefined results]". Historically, we've interpreted an escaped ordinary character as the ordinary character itself. This becomes problematic when some extensions give special meanings to an otherwise ordinary character (e.g. GNU's \b, \s, \w), meaning we may have two different valid interpretations of the same sequence. To make this easier to deal with and given that the standard calls this undefined, we should throw an error (EESCAPE) if we run into this scenario to ease transition into a state where some escaped ordinaries are blessed with a special meaning -- it will either error out or have extended behavior, rather than have two entirely different versions of undefined behavior that leave the consumer of regex(3) guessing as to what behavior will be used or leaving them with false impressions. This change bumps the symbol version of regcomp to FBSD_1.6 and provides the old escape semantics for legacy applications, just in case one has an older application that would immediately turn into a pumpkin because of an extraneous escape that's embedded or otherwise critical to its operation. This is the final piece needed before enhancing libregex with GNU extensions and flipping the switch on bsdgrep. [1] http://pubs.opengroup.org/onlinepubs/9699919799.2016edition/ PR: 229925 (exp-run, courtesy of antoine) Differential Revision: https://reviews.freebsd.org/D10510 Changes: head/contrib/netbsd-tests/lib/libc/regex/data/meta.in head/contrib/netbsd-tests/lib/libc/regex/data/subexp.in head/lib/libc/regex/Symbol.map head/lib/libc/regex/regcomp.c