We updated 10-Stable to r303474 old revision was about end of may. Afterwards we updated asterisk from 13.7.2 to 13.10.0 Everything seems to work, but since then every 8-16 hours asterisk stops procession sip requestst. Asterisk is running. asterisk -r works everything seems ok. but netstat -f inet show all sip port receive queses filling up. only way to resolv: restart.
Thank you Matthias. A few things: * Are the symptoms reproducible after downgrading to 13.7.2 ? * Are the symptoms reproducible in the latest 10-STABLE revision? * If you can provide the revision number of the previous FreeBSD version (from system or monitoring logs) that would be great * Are there any extra messages/warnings/errors in the system logs due to network resource exhaustion? * Please include (as attachments: - The netstat (and other) command outputs - Contents of your /etc/make.conf, sysctl.conf, /boot/loader.conf files (settings) For future issues, please also include a summary in the title as best you can, not just "category/port"
(In reply to Kubilay Kocak from comment #1) Ok one Thing i forgot: asterisk is running in a jail if that matters. > * Are the symptoms reproducible after downgrading to 13.7.2 ? can`t test that now. Will try to get old Version. > * Are the symptoms reproducible in the latest 10-STABLE revision? Mine ist from 29.7.16 aka 13 says ago. I could update but that would be a hassle. > * If you can provide the revision number of the previous FreeBSD version (from system or monitoring logs) that would be great FreeBSD 10.3-BETA3 #3 r296189: Mon Feb 29 12:55:38 CET 2016 > * Are there any extra messages/warnings/errors in the system logs due to > network resource exhaustion? no >* Please include (as attachments: > > - The netstat (and other) command Outputs netstat -f inet ---------------------8<------------------------- Active Internet connections Proto Recv-Q Send-Q Local Address Foreign Address (state) tcp4 0 0 pbxmx.62373 exch.xenet.de.ssh ESTABLISHED tcp4 0 0 pbxmx.45321 exch.xenet.de.ssh ESTABLISHED udp4 0 0 pbxm.wlan.tuberc.sip *.* udp4 2234 0 pbxm.phone.tuber.sip *.* udp4 0 0 pbxm.lan.tuberce.sip *.* udp4 0 0 pbxm.wlan.xenet..sip *.* udp4 1231 0 pbxm.phone.xenet.sip *.* udp4 0 0 pbxm.lan.xenet.d.sip *.* udp4 0 0 pbxmx.sip *.* udp4 0 0 192.168.3.239.sip *.* udp4 0 0 192.168.2.239.sip *.* udp4 0 0 pbxm.core.tuberc.sip *.* udp4 0 0 pbxm.core.xenet..sip *.* udp4 4233 0 pbxmx.lan.adslf1.sip *.* ---------------------8<------------------------- Internal LAN with active phones pbmx.phone.tubercel.de pbmx.phone.xenet.de Outgoing sip connection pbxmx.lan.adslf1.sip > - Contents of your /etc/make.conf, sysctl.conf, > /boot/loader.conf files (settings) make.conf on build machine sets only OBJDIRPREFIX. sysctl.conf in Jail is empty sysctl.conf on host only sets IPSEC ctls. loader.conf in jail is empty loader.conf on host: ---------------------8<------------------------- geom_stripe_load="yes" geom_mirror_load="yes" autoboot_delay="1" ---------------------8<------------------------- > For future issues, please also include a summary in the title as best you > can, not just "category/port" Ok.
Hi, I need a little more details on the asterisk setup: Are you using the PJSIP backend or the old chan_sip backend. Are you using TCP or UDP for SIP connections? And is TLS being used or not? There a re some issues in the asterisk bug tracker which could be related, for example this one: https://issues.asterisk.org/jira/browse/ASTERISK-26145 Also this could be related: http://downloads.asterisk.org/pub/security/AST-2016-005.html If you're using pjsip I'd also like to know which pjsip version. Thanks.
(In reply to Guido Falsi from comment #3) > I need a little more details on the asterisk setup: > Are you using the PJSIP backend or the old chan_sip backend. PJSIP. > Are you using TCP or UDP for SIP connections? And is TLS being used or not? only UDP is used. see netstat Output. No TLS. > There a re some issues in the asterisk bug tracker which > could be related, for example this one: > https://issues.asterisk.org/jira/browse/ASTERISK-26145 no logmessages no 500 threads seems not to be related. > Also this could be related: > http://downloads.asterisk.org/pub/security/AST-2016-005.html no tcp involved. > If you're using pjsip I'd also like to know which pjsip version. pjsip-extsrtp-2.5.1_1
(In reply to matthias+freebsd+bugzilla from comment #4) > (In reply to Guido Falsi from comment #3) > > > I need a little more details on the asterisk setup: > > Are you using the PJSIP backend or the old chan_sip backend. > > PJSIP. > > > Are you using TCP or UDP for SIP connections? And is TLS being used or not? > > only UDP is used. see netstat Output. > No TLS. Right, I should have noticed myself. > > If you're using pjsip I'd also like to know which pjsip version. > > pjsip-extsrtp-2.5.1_1 I would not bet on this, but I upgraded the pjsip port to version 2.5.5 recently, it is worth a try. I'd really also like to know if the problem disappears reverting to a previous asterisk version.
(In reply to Guido Falsi from comment #5) > > I would not bet on this, but I upgraded the pjsip port to version 2.5.5 > recently, it is worth a try. > > I'd really also like to know if the problem disappears reverting to a > previous asterisk version. I tried to switch back to 13.7.2 but was impossible. We updates to 13.10 to use "bind_rtp_to_media_address=yes" system ist multihomed and pulled all previous firewall stuff to run without that option. So just downgrading doesn`t work. Next I tried to upgrade all ports to the latest Version. pkg delete "*" pkg autoremove to remove alle installed pkgs pkg update pkg install -r asterisk13 now I have asterisk13.10.0 pjsip-2.5.5 and a *LOT* more pks. e.g. gcc-4-xxx This time if I start asterisk registrations dont work at all: [Aug 12 15:14:22] WARNING[100885] res_pjsip_outbound_registration.c: No response received from 'sip:tel.xxxxxx.de' on registration attempt to 'sip:05XXXXXXX@tel.xxxxxx.de', retrying in '60' This Error comes IMEDIATE after Registration Attempt. With tcpdump i see valid Registration Responses. after some retries asterisk vanisched completly. no logentry no Messages asterisk process ist just gone. asterisk -rrr doesnt Show up anything it just exits. restores everything to my own builded asterisk.
(In reply to matthias+freebsd+bugzilla from comment #6) > (In reply to Guido Falsi from comment #5) > > > > I would not bet on this, but I upgraded the pjsip port to version 2.5.5 > recently, it is worth a try. > > > > I'd really also like to know if the problem disappears reverting to a > > previous asterisk version. > > > I tried to switch back to 13.7.2 but was impossible. > We updates to 13.10 to use "bind_rtp_to_media_address=yes" > system ist multihomed and pulled all previous firewall stuff > to run without that option. This problem you're seeing is a little difficult to pin down. The OS was upgraded(with a biggish jump), asterisk was upgraded and both the OS/firewall and asterisk configurations have been modified in the while. The problem could have appeared in a three month time span in any of those components or even be a combination of factors. There is really no way to isolate the problem without testing each change in isolation. You could try putting your present setup on the previous FreeBSD version and see what happens, if it still hangs there it would exclude the OS upgrade as a cause, otherwise it would make me think the probem happened in The OS network stack(but my guess expressed in the next paragraph is that this is not the case). My wild guess, for what it's worth, is that you have hit a bug in asterisk+pjsip with your new configuration that the previous machine configuration did not hit, but I have no real information to corroborate this. As a sidenote I never used the bind_rtp_to_media_address=yes option. While I understand why it is desirable in your situation I > > So just downgrading doesn`t work. > > Next I tried to upgrade all ports to the latest Version. > > pkg delete "*" > pkg autoremove > > to remove alle installed pkgs > > > pkg update > pkg install -r asterisk13 > > now I have > > asterisk13.10.0 > pjsip-2.5.5 > > and a *LOT* more pks. e.g. gcc-4-xxx The default asterisk package has many options turned on and is built by default using gcc from ports. asterisk 11 and 13 can be built using clang from base. I plan to make building asterisk13 with clang the default in the near future, since in my testing it works fine. I can't say the same about asterisk11, so that port will still be built using gcc. > > This time if I start asterisk registrations dont work at all: > > [Aug 12 15:14:22] WARNING[100885] res_pjsip_outbound_registration.c: No > response received from 'sip:tel.xxxxxx.de' on registration > attempt to 'sip:05XXXXXXX@tel.xxxxxx.de', retrying in '60' > > This Error comes IMEDIATE after Registration Attempt. > > With tcpdump i see valid Registration Responses. > > after some retries asterisk vanisched completly. > > no logentry > no Messages > > asterisk process ist just gone. > > asterisk -rrr doesnt Show up anything it just exits. Looks like in your configuration the default asterisk package is crashing. This is not good news, but I don't think I'm able to diagnose this one, especially without looking in detail at your configuration. BTW the crash could also be caused by interaction between your configuration and the default asterisk port options. > > restores everything to my own builded asterisk. Did you build this asterisk using the port or by hand extracting the original source and using the asterisk provided configuration tools? If you built it using the port could you attach the output of "make showconfig"? I know that looking at debugging output is tedious and can slow down asterisk but it could be worth to try running "asterisk -r" for a day or two capturing the output after enabling debugging like this(from the asterisk wiki): core set verbose 4 core set debug 4 pjsip set logger on
(In reply to Guido Falsi from comment #7) > This problem you're seeing is a little difficult to pin down. The OS was > upgraded(with a biggish jump), asterisk was upgraded and both the OS/firewall > and asterisk configurations have been modified in the while. The Problem > could have appeared in a three month time span in any of those components or > even be a combination of factors. > > There is really no way to isolate the problem without testing each Change > in isolation. > You could try putting your present setup on the previous FreeBSD version and > see what happens, if it still hangs there it would exclude the OS upgrade as > a cause, otherwise it would make me think the probem happened in The OS > network stack(but my guess expressed in the next paragraph is that this is > not the case). I will try this. > My wild guess, for what it's worth, is that you have hit a bug in > asterisk+pjsip with your new configuration that the previous machine > configuration did not hit, but I have no real information to corroborate this. > As a sidenote I never used the bind_rtp_to_media_address=yes option. While I > understand why it is desirable in your situation I I also will try to go back to old configuration. > The default asterisk package has many options turned on and is built by > default using gcc from ports. > asterisk 11 and 13 can be built using clang from base. I plan to make > building asterisk13 with clang the default in the near future, since in my > testing it works fine. I can't say the same about asterisk11, so that port > will still be built using gcc. Very good. I personaly hate unnumbered dependencies. my own asterisk package ist build with make OPTIONS_UNSET="EXCHANGE FREETDS LDAP LUA MP3PLAYER OOH323 PORTAUDIO RADIUS SNMP GCC MYSQL PGSQL ODBC NEWT" OPTIONS_SET="SRTP BASE NCURSES" resulting in make showconfig outputting this ===> The following configuration options are available for asterisk13-13.10.0: ASTVERSION=off: Install astversion (requires bash) BACKTRACE=off: Stack backtrace support via (lib)execinfo CURL=on: Data transfer support via cURL DAHDI=on: DAHDI support EXCHANGE=off: Exchange calendar support FREETDS=off: FreeTDS library support LDAP=off: LDAP protocol support LUA=off: Lua scripting language support MP3PLAYER=off: Install MP3 Player for Music-On-Hold (mpg123) OOH323=off: ooh323 support PJSIP=on: Build the PJSIP based SIP channel PORTAUDIO=off: PortAudio library support RADIUS=off: RADIUS protocol support SNMP=off: SNMP network protocol support SPANDSP=on: SpanDSP faxing support SRTP=on: SecureRTP support SYSINFO=off: Use devel/libsysinfo to get system information XMPP=on: XMPP/GTALK support ====> Menuselect Interface Backend: you have to choose at least one of them NCURSES=on: Console (text) interface support NEWT=off: Newt User Interface ====> Encoder/Decoder (Codec) Support GSM=on: GSM codec support NEWG711=off: New G711 Codec SPEEX=on: Speex audio format support VORBIS=on: Ogg Vorbis audio codec support ====> Database Support MYSQL=off: MySQL database support ODBC=off: ODBC database backend PGSQL=off: PostgreSQL database support SQLITE2=on: SQLite 2 database support ====> Core and Music-on-Hold (MoH) Sound Files G729=off: Install G.729 format sounds ====> Compiler to use: you have to select exactly one of them BASE=on: Use base compiler (experimental) GCC=off: Build with modern GCC (from ports) ===> Use 'make config' to modify these settings > > > > This time if I start asterisk registrations dont work at all: > > > > [Aug 12 15:14:22] WARNING[100885] res_pjsip_outbound_registration.c: No > > response received from 'sip:tel.xxxxxx.de' on registration > > attempt to 'sip:05XXXXXXX@tel.xxxxxx.de', retrying in '60' > > > > This Error comes IMEDIATE after Registration Attempt. > > > > With tcpdump i see valid Registration Responses. > > > > after some retries asterisk vanisched completly. > > > > no logentry > > no Messages > > > > asterisk process ist just gone. > > > > asterisk -rrr doesnt Show up anything it just exits. > > Looks like in your configuration the default asterisk package is crashing. > > This is not good news, but I don't think I'm able to diagnose this one, > > especially without looking in detail at your configuration. > > BTW the crash could also be caused by interaction between your > configuration > and the default asterisk port options. > > > > > > restores everything to my own builded asterisk. > Did you build this asterisk using the port or by hand extracting the original > source and using the asterisk provided configuration tools? This Version an all dependencies cam from the official pkg repo. pkg install asterisk..... > you built it using the port could you attach the output of "make showconfig"? No. PKG not Port. > know that looking at debugging output is tedious and can slow down asterisk > t it could be worth to try running "asterisk -r" for a day or two capturing > e output after enabling debugging like this(from the asterisk wiki): > core set verbose 4 > core set debug 4 > pjsip set logger on For my own astersik this Shows nothing unnormal. Just no more call are processed. If i do a gbd session most threads are waiting for a mutex. I think this is a astersik/pjsip problem too. The official pkg terminates to quick to connect via asterisk console. I will try to setup a testsystem to rule out OS Version / Jailed Enviroment / Epair Interfaces etc. pp.
(In reply to matthias+freebsd+bugzilla from comment #8) (While I wait for your test results I'll ask for some clarifications) > > Very good. I personaly hate unnumbered dependencies. > > my own asterisk package ist build with So, from information you gave later, you have all asterisk dependencies installed as official binary packages, but asterisk itself is installed from a port with custom configuration, am I right? Unluckily such a setup, while working fine all the time, is not a fully supported one, there are various reasons why such a configuration can fail, first one is that the official packages are built all with default options and there can be differences with what your custom built package expects. The only supported way to have custom built binary packages is by using a tool like poudriere or synth. Depending on many factors, including the details of how you built your custom asterisk package, this could also be contributing to your problem. > > know that looking at debugging output is tedious and can slow down asterisk > > t it could be worth to try running "asterisk -r" for a day or two capturing > > e output after enabling debugging like this(from the asterisk wiki): > > > core set verbose 4 > > core set debug 4 > > pjsip set logger on > > For my own astersik this Shows nothing unnormal. > Just no more call are processed. > If i do a gbd session most threads are waiting for a mutex. Could you investigate on which mutex they are waiting for? this could give some insight of where the problem actually lays. > > I think this is a astersik/pjsip problem too. > > The official pkg terminates to quick to connect via asterisk console. > > I will try to setup a testsystem to rule out OS Version / Jailed Enviroment > / Epair Interfaces etc. pp. Thanks, this information is really necessary to understand what is going wrong.
(In reply to Guido Falsi from comment #9) > So, from information you gave later, you have all asterisk > dependencies installed as official binary packages, but asterisk > itself is installed from a port with custom configuration, > am I right? No. > Unluckily such a setup, while working fine all the time, > is not a fully supported one, there are various reasons why such a > configuration can fail, first one is that the official packages are > built all with default options and there can be differences with > what your custom built package expects. I know. > The only supported way to have custom built binary packages is by > using a tool like poudriere or synth. I use my own written "poudrie" witch does basicaly a "make package-recursive". > Depending on many factors, including the details of how you built > your custom asterisk package, this could also be contributing to > your problem. Original Setup: uptodate 10-Stable form 15.Aug hostsystem fresch installed jail for asterisk. complicated Network Setup asterisk build with Oprions: OPTIONS_UNSET="EXCHANGE FREETDS LDAP LUA MP3PLAYER OOH323 PORTAUDIO RADIUS SNMP GCC MYSQL PGSQL ODBC NEWT" OPTIONS_SET="SRTP BASE NCURSES" all depenencies fresh build. Runs fine for about 3-6hours then just irgnores incoming sip packets. (Network Queues filling up. Next Setup: Removed all Packages and installes asteris 13.10 an alle dependencies form official repository. Result: Runs for about 1 Secound. vanisches without any error. no time to connect via asterisk -r Many many other Setups. :) Last Setup: fresh 10Stable Host and Jail. simple Network Setup. self compiled asterisk: can not Register external trunk. terminates after maxtries tries. (dont know if this is supposed or not) repo sterisk: just terminates immediatly. If I remove all Trunk related entries in pjsip.conf everything seems to run stable. Next I will try to get rid on Jail enviroment.
(In reply to matthias+freebsd+bugzilla from comment #10) > > > The only supported way to have custom built binary packages is by > > using a tool like poudriere or synth. > > I use my own written "poudrie" witch does basicaly a "make > package-recursive". > Got it, thanks for clarifying. > > > Depending on many factors, including the details of how you built > > your custom asterisk package, this could also be contributing to > > your problem. > > Original Setup: > > uptodate 10-Stable form 15.Aug hostsystem > fresch installed jail for asterisk. > complicated Network Setup > > asterisk build with Oprions: > OPTIONS_UNSET="EXCHANGE FREETDS LDAP LUA MP3PLAYER OOH323 PORTAUDIO RADIUS > SNMP GCC MYSQL PGSQL ODBC NEWT" OPTIONS_SET="SRTP BASE NCURSES" > > all depenencies fresh build. > > Runs fine for about 3-6hours then just irgnores incoming sip packets. > (Network Queues filling up. Ok, this is your original report, or did you modify the asterisk configuration from there? I'm sorry but I need to understand what you mean by "complicated Network Setup", is it in contrast to some "simple network setup"? > > > > Next Setup: > > Removed all Packages and installes asteris 13.10 an alle dependencies > form official repository. > > Result: Runs for about 1 Secound. vanisches without any error. > no time to connect via asterisk -r This shows that asterisk from official repositories crashes with your setup. I have no idea why, this is definitely an asterisk bug. I'm not an asterisk developer, I just maintain the port. You will need to perform full debugging to uncover this problem, which could be caused by various causes. Unluckily I can't reproduce this crash. > > > > Many many other Setups. :) If, as I suspect it is a bug in asterisk what you need to do is perform full debugging, identifying which threads are locked and on which mutex, then you can report this information in a bug report to the asterisk developers. I can help but cannot do this in your place. You said you observed thread waiting for a mutex, could you identify which mutex and which threads? especially, in which function those threads were locked? You could need to compile asterisk with full debugging symbols and use gdb to analyse it. This is tedious, but there really is no other way. > > > Next I will try to get rid on Jail enviroment. I don't think being in a jail is involved in this.
Created attachment 173965 [details] Asterisk 13.11.0-RC1 patch I'm attaching this patch which updates the port to a newer Release Candidate. You could try with this version, as usual I would not bet on it, but maybe it works better for you. you can find a changelog here: http://www.asterisk.org/downloads/news
(In reply to Guido Falsi from comment #11) > Ok, this is your original report, or did you modify the asterisk > configuration from there? I'm sorry but I need to understand what > you mean by "complicated Network Setup", is it in contrast to some > "simple network setup"? Ok. original "complicated" setup. about 8 local LANS (10.x.x.x / 192.168.x.x) with local phones on ist. 1 local lan 192.168.14.x with a Gateway to the real world. where the sip data schould flow. 1 WAN with external IP and Gatway to the world. running Setup with 13.7 asterisk the Default route pointed to the 192.168.14.x Gateway. Traffik got nated to secound external IP. worked. But! I could not connect Phones via WAN Iface because RTP Traffik binds to 0.0.0.0 and is routet out via secound external IP. to solve this i updated to 13.10 with new Option bind_rtp_to_media_adress. then 1 Changes the default route WAN Gateway and forwarded all traffik from 192.168.4.x to the old default gate. This worked but only for 3 to 6 hours an then stopped. (Prozessing SIP PAckets asterisk sill running) to nail this down. The simple network only consists of a single Interface ( with nated IP 182.168.14.x) asterisk[192.168.14.6]---->[192.168.14.254]natGateway[official external IP]---> Internet only pjsip.conf entries to Register a trunk. now it runs but can not register. A Sip Register is going out A SIP Unathorized is coming back. But then no attempt ist made to Register with authorisation Header. right now am out of clues. I will set this aside till tomorrow.
Hi all, finally this turned out to be an asterisk pjsip bug https://issues.asterisk.org/jira/browse/ASTERISK-26186 This bug is hit only in multihomed enviroments. And it is hit only from time to time. I moved back to chan_sip and now every thing works as expected. I think this Report can be closed as this is not freebsd related. Many thanks for your Support.
Thanks, closing PR as it's an upstream bug, not a bug in the port. Nothing to merge to quarterly at present.