Bug 198089 - [patch update] www/squid 3.4.12 -> 3.5.3
Summary: [patch update] www/squid 3.4.12 -> 3.5.3
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Kurt Jaeger
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-02-28 07:49 UTC by Kurt Jaeger
Modified: 2015-06-07 21:24 UTC (History)
2 users (show)

See Also:


Attachments
new port (114.33 KB, text/plain)
2015-02-28 07:49 UTC, Kurt Jaeger
no flags Details
squid35-11.03.15.shar (114.51 KB, text/plain)
2015-03-11 09:01 UTC, timp87
no flags Details
squid35-20.03.15.shar (114.96 KB, text/plain)
2015-03-20 08:32 UTC, timp87
no flags Details
patch for www/squid (30.22 KB, patch)
2015-03-30 09:59 UTC, timp87
no flags Details | Diff
patch for www/squid with fixed build on 8 and 9 (31.20 KB, patch)
2015-04-07 09:04 UTC, timp87
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Comment 1 Kurt Jaeger freebsd_committer 2015-02-28 11:07:54 UTC
build testing: works on 10.1a, fails on 9.3a, 8.4i.

poudriere build logs can be found at

http://people.freebsd.org/~pi/logs/www__squid35*

Older builds are with a custom config, newer builds with the generic config.
Comment 2 timp87 2015-03-11 09:01:47 UTC
Created attachment 154185 [details]
squid35-11.03.15.shar

Difference between previous shar version:
   - fix install on FreeBSD 9 and 8, I hope;
   - disable gnutls. It affects only squidclient.



The root of build problem is NIS auth helper.
I'm not a smart person here, so, please, don't hit me hard =)

The squid's configure script has code in 31845-31865 lines which decides if we can build NIS helper:
...
BUILD_HELPER="NIS"
for ac_header in sys/types.h rpc/rpc.h rpcsvc/ypclnt.h rpcsvc/yp_prot.h crypt.h
do :
  as_ac_Header=`$as_echo "ac_cv_header_$ac_header" | $as_tr_sh`
ac_fn_cxx_check_header_compile "$LINENO" "$ac_header" "$as_ac_Header" "
#if HAVE_RPC_RPC_H
#include <rpc/rpc.h>
#endif


"
if eval test \"x\$"$as_ac_Header"\" = x"yes"; then :
  cat >>confdefs.h <<_ACEOF
#define `$as_echo "HAVE_$ac_header" | $as_tr_cpp` 1
_ACEOF

else
  BUILD_HELPER=""
fi
...

There is a list of headers which must be tried to compile. What's wrong here?
   1. rpcsvc/yp_prot.h can't be compiled on FreeBSD 8, 9 and 10 because of this active PR https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=188247.
Configure script tells us in config.log:
usr/include/rpcsvc/yp_prot.h:71:15: error: redeclaration of C++ built-in type 'bool'
typedef u_int bool;
              ^
   2. crypt.h can't be compiled on FreeBSD < 10. Yes, I pointed it to right place, to rpcsvc/crypt.h, see the files/patch-configure_crypt.h.
Configure script tells us in config.log:
/usr/include/rpcsvc/crypt.h:30: error: declaration of 'des_dir desargs::des_dir'
/usr/include/rpcsvc/crypt.h:20: error: changes meaning of 'des_dir' from 'typedef enum des_dir des_dir'
/usr/include/rpcsvc/crypt.h:31: error: declaration of 'des_mode desargs::des_mode'
/usr/include/rpcsvc/crypt.h:26: error: changes meaning of 'des_mode' from 'typedef enum des_mode des_mode'


Why it worked in squid 3.4 branch? The result of these header checks wasn't handled properly. That's all. So I just made the same - returned that broken behaviour, see the files/patch-configure-NIS.



TODO:
   - rewrite rc script to support running multiple instances (http://wiki.squid-cache.org/MultipleInstances);
   - provide single_option to build squid with different kerberos implementations (base, security/heimdal, security/krb5).



Active problems, i.e. why we can't add it to ports tree right now:
   1. In some difficult reproducible cases negotiate_kerberos_auth crashes and squid starts and stops in loop.
   2. Sometimes squid crashes - "kernel: pid 44436 (squid), uid 100: exited on signal 6 (core dumped)".

I'm testing squid-3.5 on FreeBSD 10.1-RELEASE amd64.

P.S. I'd highly appreciate any help and ideas! I don't have much time and skills to investigate the problems, but I'm doing my best.
Comment 3 Kurt Jaeger freebsd_committer 2015-03-11 09:38:36 UTC
Testbuild works for 10.1a, 9.3a, fails on 8.4i, see

http://people.freebsd.org/~pi/logs/www__squid35-84i-1426065484.txt

I will look into the 8.4i build issue.
Comment 4 timp87 2015-03-11 09:48:56 UTC
(In reply to Kurt Jaeger from comment #3)
I could build it on fresh FreeBSD 8.4-RELEASE-p24 amd64.
Comment 5 timp87 2015-03-20 08:32:18 UTC
Created attachment 154557 [details]
squid35-20.03.15.shar

It's a sort of release candidate =)

Differences between previous shar version:
- squid can be built with different kerberos implementations now.
- active problems, which were mentioned previously, are resolved (see file/patch-bug4213 and file/patch-bug4190). One patch is from upstream and another is just a revert of buggy patch from upstream.
- Dennis Glatting's strlen() patches was removed. Looks like squid works well without them. *Need more people to test it to make sure it's true.

Only one thing left: rewrite rc script to support running multiple instances.
But I think I'll (or someone) realize it in future. Too many changes for one time.
Comment 6 Kurt Jaeger freebsd_committer 2015-03-24 12:46:38 UTC
Pavel Timofeev found

http://bugs.squid-cache.org/show_bug.cgi?id=4030

and put in the bugreport with "having krb5 from the ports but building
against krb5 from the base".
Comment 7 timp87 2015-03-30 09:59:46 UTC
Created attachment 154988 [details]
patch for www/squid

This time it's a patch for existing www/squid. It's squid-3.5.3.
I hope I did it right.

Looking forward to see it in ports tree!
Comment 8 Kurt Jaeger freebsd_committer 2015-03-31 19:52:16 UTC
I found a fix for the krb5 issue in src/peer_proxy_negotiate_auth.cc,

see http://bugs.squid-cache.org/show_bug.cgi?id=4030

but 3.5.3 dies with other problems on 9.3a, 8.4i in poudriere, now.
Comment 9 timp87 2015-04-02 09:30:10 UTC
(In reply to Kurt Jaeger from comment #8)

It fails on 8.4 and 9.3 because I started to use /usr/ports/Mk/Uses/gssapi.mk to select different krb implementation in port's Makefile. Before that it was able to build. I tried an old port's Makefile - it builds.
Comment 10 timp87 2015-04-06 15:59:52 UTC
(In reply to Kurt Jaeger from comment #8)
Well, I just realized that solution you proposed isn't good.
As I understand, problem occurs because squid build process takes right lib from right place, but takes wrong header in the same time. And it's bad.
I tried to figure out how to fix it in other way. No luck.
CXXFLAGS is set to "-I/usr/include -I/usr/local/include" at the same time and preprocessor (don't know why) takes header only from the second path. Don't know how to set the CXXFLAGS only for krb headers.
Comment 11 timp87 2015-04-07 09:04:43 UTC
Created attachment 155295 [details]
patch for www/squid with fixed build on 8 and 9

New patch. Build on 8 and 9 is fixed.

As for problem with "build with base kerberos when security/krb5 is installed from ports" I couldn't find a good fix. Looks like it's really hard do make it.

Everything would simple if security/krb5 installed into special separate directory, like security/heimdal does. I mean not to /usr/local/.

Sorry.
Comment 12 Kurt Jaeger freebsd_committer 2015-04-13 17:53:33 UTC
testing@work
Comment 13 Kurt Jaeger freebsd_committer 2015-04-13 18:28:03 UTC
Builds fine on 10.1a, 9.3a, 8.4i.

Will wait 24h for other testers, then commit.
Comment 14 commit-hook freebsd_committer 2015-04-14 16:34:52 UTC
A commit references this bug:

Author: pi
Date: Tue Apr 14 16:34:40 UTC 2015
New revision: 384005
URL: https://svnweb.freebsd.org/changeset/ports/384005

Log:
  www/squid: 3.4.12 -> 3.5.3

  As it is written on http://www.squid-cache.org/Versions/ for 3.4.x
  versions: they are no longer intended for general use in new installations.
  So this is a big upgrade.

  Changes to squid-3.5.3 (28 Mar 2015):

  - Regression Bug 4213: negotiate_kerberos_auth: freeing non-dynamic memory
  - Regression Bug 4206: Incorrect connection close on expect:100-continue
  - Bug 4204: ./configure does not abort when required helpers cannot be built
  - Bug 3805: support shared memory on MacOS X in Mem::IPC::Segment
  - Bug 2907: high CPU usage on CONNECT when using delay pools
  - basic_getpwnam_auth: fail authentication on crypt() failures
  - basic_nis_auth: fail authentication on crypt() failures
  - ext_kerberos_ldap_group_acl: Heimdal support improvements
  - ext_wbinfo_group_acl: Perl 5.20 support
  - ... and several compile issues

  Changes to squid-3.5.2 (18 Feb 2015):

  - Regression Bug 4176: Digest auth too many helper lookups
  - Regression Bug 4180: not-fully-initialized data member in ACLUserData
  - Bug 4172: Solaris broken krb5-config
  - Bug 4073: Cygwin compile errors
  - Bug 3919: remove several never-true / never-false comparisons
  - HTTPS: Add missing root CAs when validating chains that passed
    internal checks
  - Fix some cbdataFree related memory leaks
  - Quieten CBDATA 'leak' messages
  - Set SNI information in transparent bumping mode
  - negotiate_kerberos_auth: fix krb5.conf backward compatibility
  - Fix memory leaks in cachemgr.cgi URL parser
  - Fix sslproxy_options in peek-and-splice mode
  - ... and fix several portability and build issues
  - ... and some documentation updates
  - ... and all fixes from squid 3.4.11

  Changes to squid-3.5.1 (13 Jan 2015):

  - Fix handling of invalid SSL server certificates when splicing connections
  - basic_smb_lm_auth: Simplified MSNT basic auth helper
  - squidclient: Fix -A and -P options
  - ... and several portability fixes
  - ... and all fixes from squid 3.4.11
  - ... and a lot of documentation updates

  Changes to squid-3.5.0.4 (21 Dec 2014):

  - Bug 3826: pt 2: Provide a systemd .service file for Squid
  - Support http_access denials of SslBump "peeked" connections.
  - Fix DONT_VERIFY_DOMAIN ssl flag
  - Fix peek-and-splice mode: certificate validation for domain mismatched errors
  - negotiate_kerberos_auth: MEMORY keytab and replay cache support
  - ... and some documentation updates
  - ... and a large amount of code polishing (non-logic changes)

  Changes to squid-3.5.0.3 (09 Dec 2014):

  - Bug 4146: workaround SSL Bump crash on Linux
  - Bug 4135: Support \-escaped characters in regex patterns
  - Bug 4131: SIGSEGV at store.cc:962 content_length > store_maxobjsize
  - Fix delay_parameters parsing
  - HTTP/2: handle 'PRI' method found in HTTP/1.x traffic
  - ... and all changes from squid 3.4.10
  - ... and a lot of documentation updates

  Changes to squid-3.5.0.2 (31 Oct 2014):

  - Fix FTP socket opening during reconfigure
  - ... and all changes from 3.4.9
  - ... and some build errors in rarely used code
  - ... and several documentation updates

  Changes to squid-3.5.0.1 (17 Oct 2014):

  - Port from 2.7: redirector and logging urlgroup feature
  - Bug 4093: source-maintenance.sh bad perl -i option
  - Bug 3608: per-service name for workers UDS sockets
  - Bug 2554: 32-bit wrap in AUFS counters
  - Bug 1961 pt1: URL handling redesign
  - Bug 1202 pt1: documentation for refresh_pattern algorithms
  - Update Squid boilerplate copyright/license
  - Update the http(s)_port directives protocol= parameter
  - Update forward_max_tries to permit 25 server paths
  - Update Kerberos library detection and build options
  - Support ACLs on ftp_epsv directive
  - Support >32KB objects in cache_dir rock storage
  - Support client connection annotation by helpers via clt_conn_tag=TAG
  - Support native FTP Relay
  - Support libgnugss Kerberos library
  - Support libecap v1.0
  - Support SSL Peek and Splice feature
  - Support receiving PROXY protocol version 1 and 2
  - Replace --enable-ssl build option with --with-openssl
  - Enable -n service name command line option for all Squid builds
  - Enable ICAP client by default
  - Fix configuration file parsing bugs, related to quoted strings
  - Fix Windows MinGW build errors
  - Fix multiple TCP outgoing TOS/DiffServ bugs
  - Fix Cygwin /etc/resolv.conf parsing
  - Fix crash when sending %ssl::cert_subject to external ACL w/o certificate
  - Fix crash reading malformed config files
  - Send selected SSL version and cipher to the certificate validation helper
  - Validate server certificates without bumping
  - Add zero-copy string buffer support
  - Add automated squid.conf parser testing with squid -k parse
  - Add adaptation_service ACL
  - Add logformat code %tS to log transaction start time
  - Add logformat code %>rd to log client URL domain name
  - Add key_extras to proxy authentication
  - Add url_rewrite_extras and store_id_extras directives
  - Add send_hit and store_miss directives
  - Add collapsed_forwarding directive
  - Add sslproxy_cert_sign_hash directive
  - Add SMP SSL session cache
  - Add cache_peer standby connections
  - Add helper ext_delayer_acl
  - Add TCP_TUNNEL log code for CONNECT tunnels which are not SSL-bumped
  - Add BUILDCXX and BUILDCXXFLAGS configure options for cross-compile
  - Remove COSS storage in favour of Rock storage
  - Remove dnsserver and external DNS helper API in favour of mDNS
  - Remove broken mallinfo() accounting and memory tracing
  - Remove hierarchy_stoplist in favour of always_direct
  - Deprecate tag ACL type in favour of note ACL type
  - Deprecate urlgroup feature in favour of note ACL type
  - HTTP/1.1: method names are case-sensitive
  - HTTP/1.1: register new headers from RFC 723x
  - squidclient: polish and update help display
  - squidclient: support TLS with GnuTLS 3.1.5+
  - squidclient: support verbosity levels
  - squidclient: --ping mode module support
  - url_fake_rewrite: support concurrency
  - storeid_file_rewrite: support concurrency
  - digest_file_auth: support concurrency
  - digest_edirectory_auth: support concurrency
  - digest_ldap_auth: support concurrency
  - ... and many error page translation updates
  - ... and much code cleanup and polishing

  PR:		198089
  Submitted by:	Pavel Timofeev <timp87@gmail.com>
  Tested by:	John Marshall <john.marshall@riverwillow.com.au>

Changes:
  head/www/squid/Makefile
  head/www/squid/distinfo
  head/www/squid/files/extra-patch-build-8-9
  head/www/squid/files/patch-bug4190
  head/www/squid/files/patch-compat_Makefile.in
  head/www/squid/files/patch-compat_compat.h
  head/www/squid/files/patch-compat_strlen.c
  head/www/squid/files/patch-configure
  head/www/squid/files/patch-configure_NIS
  head/www/squid/files/patch-configure_crypt.h
  head/www/squid/files/patch-src-cf.data.pre
  head/www/squid/pkg-plist
Comment 15 Kurt Jaeger freebsd_committer 2015-06-07 21:24:22 UTC
See PR 200070 for other updates. Closing this ticket now.