Bug 212321

Summary: www/trafficserver: Crashes (on 10.3-RELEASE-p7) with Signal 11: Segmentation fault
Product: Ports & Packages Reporter: David Brodin <brodin>
Component: Individual Port(s)Assignee: Hung-Yi Chen <gaod>
Status: Closed Overcome By Events    
Severity: Affects Only Me CC: gaod, w.schwarzenfeld
Priority: --- Keywords: crash, needs-patch, needs-qa
Version: LatestFlags: bugzilla: maintainer-feedback? (gaod)
Hardware: Any   
OS: Any   
URL: https://issues.apache.org/jira/browse/TS-4816
See Also: https://issues.apache.org/jira/browse/TS-4816

Description David Brodin 2016-09-02 06:52:46 UTC 
Hi, 

We are running
[root@<machine> ~]# pkg info | grep traff
trafficserver-6.2.0            Fast, scalable and extensible HTTP proxy server

on one machine:
[root@<machine> ~]# uname -a
FreeBSD portos.sr.se 10.3-RELEASE-p7 FreeBSD 10.3-RELEASE-p7 #0: Thu Aug 11 18:38:15 UTC 2016     root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64

and ATS is crashing at least once a day with the following error in manager.log:
[Sep  1 15:27:41.613] Manager {0x804006400} ERROR: [LocalManager::sendMgmtMsgToProcesses] Error writing message
[Sep  1 15:27:41.613] Manager {0x804006400} ERROR: <MgmtUtils.cc:289 (mgmt_elog)>  (last system error 32: Broken pipe)
[Sep  1 15:27:41.929] Manager {0x804006400} ERROR: [LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 11: Segmentation fault
[Sep  1 15:27:41.929] Manager {0x804006400} ERROR: [Alarms::signalAlarm] Server Process was reset
[Sep  1 15:27:42.934] Manager {0x804006400} NOTE: [LocalManager::startProxy] Launching ts process
[Sep  1 15:27:42.949] Manager {0x804006400} NOTE: [LocalManager::pollMgmtProcessServer] New process connecting fd '15'
[Sep  1 15:27:42.949] Manager {0x804006400} NOTE: [Alarms::signalAlarm] Server Process born
[Sep  2 06:58:03.921] Manager {0x804006400} ERROR: [LocalManager::sendMgmtMsgToProcesses] Error writing message
[Sep  2 06:58:03.921] Manager {0x804006400} ERROR: <MgmtUtils.cc:289 (mgmt_elog)>  (last system error 32: Broken pipe)
[Sep  2 06:58:04.275] Manager {0x804006400} ERROR: [LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 11: Segmentation fault
[Sep  2 06:58:04.275] Manager {0x804006400} ERROR: [Alarms::signalAlarm] Server Process was reset
[Sep  2 06:58:05.292] Manager {0x804006400} NOTE: [LocalManager::startProxy] Launching ts process
[Sep  2 06:58:05.308] Manager {0x804006400} NOTE: [LocalManager::pollMgmtProcessServer] New process connecting fd '14'
[Sep  2 06:58:05.308] Manager {0x804006400} NOTE: [Alarms::signalAlarm] Server Process born

And from messages:
Sep  2 06:58:03 <machine> traffic_manager[5604]: {0x804006400} ERROR: [LocalManager::sendMgmtMsgToProcesses] Error writing message
Sep  2 06:58:03 <machine> kernel: pid 6680 (traffic_server), uid 80: exited on signal 11
Sep  2 06:58:03 <machine> traffic_manager[5604]: {0x804006400} ERROR: <MgmtUtils.cc:289 (mgmt_elog)>  (last system error 32: Broken pipe)
Sep  2 06:58:04 <machine> traffic_cop[5603]: cannot find traffic_server [1]
Sep  2 06:58:04 <machine> traffic_manager[5604]: {0x804006400} ERROR: [LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 11: Segmentation fault
Sep  2 06:58:04 <machine> traffic_manager[5604]: {0x804006400} ERROR: [Alarms::signalAlarm] Server Process was reset
Sep  2 06:58:08 <machine> traffic_server[9951]: NOTE: --- traffic_server Starting ---
Sep  2 06:58:08 <machine> traffic_server[9951]: NOTE: traffic_server Version: Apache Traffic Server - traffic_server - 6.2.0 - (build # 083112 on Aug 31 2016 at 12:51:58)
Sep  2 06:58:08 <machine> traffic_server[9951]: NOTE: RLIMIT_NOFILE(8):cur(190297),max(190297)

This of course alleviates the memory leak we also are experiencing in v6.2.0, but I have already created a PR for that earlier :)

BR,
David
Comment 1 David Brodin 2016-09-03 15:34:11 UTC 
Just found this in the traffic.out-logfile:
traffic_server[TrafficManager] ==> Cleaning up and reissuing signal #15
: Terminated
traffic_server: Terminatedtraffic_servertraffic_servertraffic_server: Segmentation fault
traffic_server - STACK TRACE:
0x4af409 <_Z19crash_logger_invokeiP9__siginfoPv+0x69> at /usr/local/bin/traffic_server
0x802735b37 <pthread_sigmask+0x507> at /lib/libthr.so.3
0x80273522c <pthread_getspecific+0xe1c> at /lib/libthr.so.3
getpeereid -> 0 (54, Connection reset by peer)[TrafficManager] ==> Cleaning up and reissuing signal #15
traffic_server: Terminated
traffic_server: Terminated
traffic_server: using root directory '/usr/local'
[TrafficManager] ==> signal #15
traffic_server: Segmentation fault
traffic_server - STACK TRACE:
0x4af409 <_Z19crash_logger_invokeiP9__siginfoPv+0x69> at /usr/local/bin/traffic_server
0x802735b37 <pthread_sigmask+0x507> at /lib/libthr.so.3
0x80273522c <pthread_getspecific+0xe1c> at /lib/libthr.so.3
traffic_server: Segmentation fault
traffic_server - STACK TRACE:
0x4af409 <_Z19crash_logger_invokeiP9__siginfoPv+0x69> at /usr/local/bin/traffic_server
0x802735b37 <pthread_sigmask+0x507> at /lib/libthr.so.3
0x80273522c <pthread_getspecific+0xe1c> at /lib/libthr.so.3
traffic_server: Terminated
traffic_server: Terminated
traffic_server: Terminated
traffic_server: Terminated
traffic_server: Segmentation fault
traffic_server - STACK TRACE:
0x4af409 <_Z19crash_logger_invokeiP9__siginfoPv+0x69> at /usr/local/bin/traffic_server
0x802735b37 <pthread_sigmask+0x507> at /lib/libthr.so.3
0x80273522c <pthread_getspecific+0xe1c> at /lib/libthr.so.3
traffic_server: Segmentation fault
traffic_server - STACK TRACE:
0x4af409 <_Z19crash_logger_invokeiP9__siginfoPv+0x69> at /usr/local/bin/traffic_server
0x802735b37 <pthread_sigmask+0x507> at /lib/libthr.so.3
0x80273522c <pthread_getspecific+0xe1c> at /lib/libthr.so.3
traffic_server: Segmentation fault
traffic_server - STACK TRACE:
0x4af409 <_Z19crash_logger_invokeiP9__siginfoPv+0x69> at /usr/local/bin/traffic_server
0x802735b37 <pthread_sigmask+0x507> at /lib/libthr.so.3
0x80273522c <pthread_getspecific+0xe1c> at /lib/libthr.so.3
traffic_server: Segmentation fault
traffic_server - STACK TRACE:
0x4af409 <_Z19crash_logger_invokeiP9__siginfoPv+0x69> at /usr/local/bin/traffic_server
0x802735b37 <pthread_sigmask+0x507> at /lib/libthr.so.3
0x80273522c <pthread_getspecific+0xe1c> at /lib/libthr.so.3
traffic_server: Segmentation fault
traffic_server - STACK TRACE:
0x4af409 <_Z19crash_logger_invokeiP9__siginfoPv+0x69> at /usr/local/bin/traffic_server
0x802735b37 <pthread_sigmask+0x507> at /lib/libthr.so.3
0x80273522c <pthread_getspecific+0xe1c> at /lib/libthr.so.3
Comment 2 Walter Schwarzenfeld 2018-09-03 14:38:20 UTC 
Is this still relevant (10.3. is EOL)?
Comment 3 David Brodin 2018-09-03 18:24:58 UTC 
No idea, we are moving away from FreeBSD. Not just because of this though.
I won't have the time to test a later version of FreeBSD.