Bug 215882

Summary: [exp-run] always set LANG=C and LC_ALL=C
Product: Ports & Packages Reporter: Tijl Coosemans <tijl>
Component: Ports FrameworkAssignee: Tijl Coosemans <tijl>
Status: Closed FIXED    
Severity: Affects Only Me CC: gerald, portmgr, ports-bugs
Priority: ---    
Version: Latest   
Hardware: Any   
OS: Any   
See Also: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=226636
Bug Depends on:    
Bug Blocks: 212624, 215792    
Attachments:
Description Flags
patch
none
patch2 none

Description Tijl Coosemans freebsd_committer 2017-01-08 21:32:55 UTC
Created attachment 178636 [details]
patch

The output of tools like awk, sort, tr,... depends on the current locale set by the user.  The attached patch adds LANG=C and LC_ALL=C at the beginning of bsd.port.mk and exports them so all commands are executed with the C locale.

It also introduces USE_LOCALE=<locale> that adds LANG=<locale> and LC_ALL=<locale> to CONFIGURE_ENV and MAKE_ENV so upstream build systems can be executed with a different locale (e.g. USE_LOCALE=en_US.UTF-8).
Comment 1 Antoine Brodin freebsd_committer 2017-01-15 21:51:22 UTC
See also bug #216124
Comment 2 Gerald Pfeifer freebsd_committer 2017-01-15 22:02:05 UTC
Of course, after I finally get to submit the patch which I had been
testing since fall, Tijl submitted this just a week earlier. :-)

This patch is quite more extensive than mine, and I intentionally only
used LC_COLLATE and in a more limited environment.

Do we really want/need to globally set LANG and the other LC_s ?
Comment 3 Tijl Coosemans freebsd_committer 2017-01-16 14:05:32 UTC
(In reply to Gerald Pfeifer from comment #2)
Yes we want the other LC_* too.  Bug 215792 requires LC_NUMERIC=C.  Use of character classes with tr(1) requires LC_CTYPE=C.  Consistent output from date(1) requires LC_TIME=C.  These are all handled at once by setting LC_ALL=C.  

LANG is used by setlocale(3) as default value for LC_* variables that aren't set, so normally it isn't used when LC_ALL is set, but I've seen patches in the ports tree that show there's code out there that looks at LANG directly so it's safer to set it as well.

I also think we need to set these globally (with .export) at the beginning of bsd.port.mk to rule out any locale problems in the ports tree itself.  The only commands not captured by this are != assignments before any inclusion of bsd.port.*mk.  I now see I forgot about bsd.port.subdir.mk.  Maybe LANG/LC_ALL should be set there as well?
Comment 4 Antoine Brodin freebsd_committer 2017-01-16 18:18:15 UTC
There is 1 new failure:

+ {"origin"=>"mail/wanderlust", "phase"=>"build", "errortype"=>"???"}

Failure log:

http://package23.nyi.freebsd.org/data/110amd64-default-PR215882/2017-01-16_06h31m26s/logs/errors/wanderlust-emacs25-2.14.0_17.log
Comment 5 Tijl Coosemans freebsd_committer 2017-01-16 20:02:29 UTC
Created attachment 178969 [details]
patch2

Attempt to fix mail/wanderlust.
Comment 6 Antoine Brodin freebsd_committer 2017-01-18 12:08:46 UTC
Exp-run looks fine
Comment 7 commit-hook freebsd_committer 2017-01-18 13:21:38 UTC
A commit references this bug:

Author: tijl
Date: Wed Jan 18 13:20:39 UTC 2017
New revision: 431796
URL: https://svnweb.freebsd.org/changeset/ports/431796

Log:
  The output of tools like awk, date, sort, tr,... depends on the current
  locale set by the user.  Add LANG=C and LC_ALL=C at the beginning of
  bsd.port.mk and export them so all commands are executed with the C locale.
  LC_ALL=C overrides all other LC_* variables.  LANG is used by setlocale(3)
  as default value for LC_* variables, so normally it isn't used when LC_ALL
  is set, but there's code out there that looks at LANG directly so it's safer
  to set it as well.  The only commands not captured by this are !=
  assignments before any inclusion of bsd.port.*mk.

  Introduce USE_LOCALE=<locale> that adds LANG=<locale> and LC_ALL=<locale> to
  CONFIGURE_ENV and MAKE_ENV so upstream build systems can be executed with a
  different locale (e.g. USE_LOCALE=en_US.UTF-8).

  PR:		215882
  Exp-run by:	antoine
  Approved by:	portmgr (antoine)

Changes:
  head/Mk/Scripts/check-stagedir.sh
  head/Mk/Uses/elixir.mk
  head/Mk/Uses/gem.mk
  head/Mk/bsd.commands.mk
  head/Mk/bsd.port.mk
  head/archivers/libzip/Makefile
  head/audio/denemo/Makefile
  head/audio/libamrnb/Makefile
  head/audio/libamrwb/Makefile
  head/base/gcc/Makefile
  head/cad/astk-serveur/Makefile
  head/cad/jspice3/files/patch-build
  head/chinese/big5width/Makefile
  head/chinese/big5width/pkg-plist
  head/devel/cutter/Makefile
  head/devel/libc++/Makefile
  head/devel/libedit/Makefile
  head/devel/libopenbsd/Makefile
  head/devel/libpci/Makefile
  head/devel/libublio/Makefile
  head/devel/powerpc64-gcc/Makefile
  head/devel/riscv64-gcc/Makefile
  head/devel/tcllib/Makefile
  head/dns/maradns/Makefile
  head/editors/emacs/Makefile
  head/editors/emacs-devel/Makefile
  head/editors/libreoffice/Makefile
  head/editors/libreoffice4/Makefile
  head/editors/openoffice-4/Makefile
  head/editors/openoffice-devel/Makefile
  head/editors/tea/Makefile
  head/editors/yudit/Makefile
  head/emulators/tme/Makefile
  head/ftp/bsdftpd-ssl/files/patch-contrib-libedit-makelist
  head/ftp/curl/Makefile
  head/ftp/ftpcopy/files/patch-src_typesize.sh
  head/ftp/pure-ftpd/Makefile
  head/games/blackjackclient/Makefile
  head/games/doomsday/Makefile
  head/games/jchessboard/Makefile
  head/games/openbor/Makefile
  head/games/quake-data/Makefile
  head/german/hunspell/Makefile
  head/graphics/Coin/Makefile
  head/graphics/jogl/Makefile
  head/graphics/libimg/Makefile
  head/japanese/skk-jisyo/Makefile
  head/java/bootstrap-openjdk/Makefile.update
  head/java/icedtea-web/Makefile
  head/java/jakarta-commons-httpclient/Makefile
  head/java/jfreechart/Makefile
  head/java/openjdk6/Makefile
  head/java/openjdk7/Makefile
  head/java/openjdk8/Makefile
  head/lang/elixir/Makefile
  head/lang/gauche/Makefile
  head/lang/gcc/Makefile
  head/lang/gcc48/Makefile
  head/lang/gcc49/Makefile
  head/lang/gcc5/Makefile
  head/lang/gcc5-devel/Makefile
  head/lang/gcc6/Makefile
  head/lang/gcc6-devel/Makefile
  head/lang/ghc/Makefile
  head/lang/ghc/bsd.cabal.mk
  head/lang/mono/Makefile
  head/lang/perl5-devel/Makefile
  head/lang/perl5.20/Makefile
  head/lang/perl5.22/Makefile
  head/lang/perl5.24/Makefile
  head/lang/rubinius/Makefile
  head/lang/ruby21/Makefile
  head/lang/ruby22/Makefile
  head/lang/ruby23/Makefile
  head/lang/ruby24/Makefile
  head/lang/tcl84/Makefile
  head/lang/tcl85/Makefile
  head/lang/tcl86/Makefile
  head/lang/tcl87/Makefile
  head/lang/yap-devel/Makefile
  head/mail/mew/files/patch-info__Makefile
  head/mail/wanderlust/files/patch-Makefile
  head/mail/wanderlust-devel/files/patch-Makefile
  head/math/geonext/Makefile
  head/math/jakarta-commons-math/Makefile
  head/math/jakarta-commons-math3/Makefile
  head/math/scilab/Makefile
  head/math/scilab-toolbox-sivp/Makefile
  head/math/scilab-toolbox-swt/Makefile
  head/multimedia/bino/Makefile
  head/multimedia/libvpx/Makefile
  head/multimedia/miro/Makefile
  head/net-mgmt/smokeping/Makefile
  head/net-p2p/mldonkey/Makefile
  head/net-p2p/vuze/Makefile
  head/news/cnews/Makefile
  head/print/pdftk/Makefile
  head/science/colt/Makefile
  head/security/i2p/Makefile
  head/security/isakmpd/Makefile
  head/security/py-dfvfs/Makefile
  head/security/py-plaso/Makefile
  head/shells/fd/Makefile
  head/sysutils/cdrdao/Makefile
  head/sysutils/pciutils/Makefile
  head/sysutils/sformat/Makefile
  head/textproc/linuxdoc-tools/Makefile
  head/textproc/msort/Makefile
  head/textproc/te-hunspell/Makefile
  head/www/jericho-html/Makefile
  head/www/tdom/Makefile
  head/www/tinymce/Makefile
  head/www/trac-tags/Makefile
  head/www/trac-wysiwyg/Makefile
  head/www/websh/Makefile
  head/www/zenphoto/Makefile
  head/x11-fonts/fontconfig/Makefile
  head/x11-toolkits/Xmt/Makefile
  head/x11-toolkits/open-motif/Makefile
  head/x11-toolkits/p5-Gtk2/Makefile
Comment 8 Tijl Coosemans freebsd_committer 2017-01-18 13:22:14 UTC
*** Bug 216124 has been marked as a duplicate of this bug. ***