Created attachment 253285 [details] SSH AT Commands Hi! I am using an intel N100 CPU config, with 16GB ram and 256 SSD, with OPNSense. The next issue I have it for about 3-4 weeks, after OPNSense released a new version. They said that the issue it is not from OPNSense, insteand, could be from FreeBSD, with the recommendation to fire a bug report here. FreeBSD version 14.1-RELEASE-p3 The issue is that my Quectel EP06-E it is not working anymore. I tried with USB adapter and directly to Mini PCI port, still not working. This is what I get into the OPNSense logs: 2024-08-12T14:18:23 Informational ppp [opt1_link0] Link: reconnection attempt 148 in 3 seconds 2024-08-12T14:18:23 Informational ppp [opt1_link0] LCP: Down event 2024-08-12T14:18:23 Informational ppp [opt1_link0] Link: DOWN event 2024-08-12T14:18:23 Informational ppp [opt1_link0] MODEM: chat script failed 2024-08-12T14:18:23 Informational ppp [opt1_link0] CHAT: The modem is not responding to "AT" at MomCmd: labeell. And the attachment is to see what I get with AT commands through SSH. Any solution to this? Thank you in advance!
I can't find a hint on why Quectel modems would stop behaving like they did on FreeBSD 13. We've recently tested Sierra just fine and the issue doesn't appear to be wide spread amongst modems. Tried reverting https://cgit.freebsd.org/src/commit/?id=d3a83456e1e3 but that didn't help. One thing of note in this commit's case is that it breaks the ascending order by device ID. What is absolutely out of whack is logging duplicated characters at the end of the comamnd: > CHAT: The modem is not responding to "AT" at MomCmd: labeell. > CHAT: line 358: label "MomIdentGeneriicc" not found Cheers, Franco
I'm experiencing the same issue. My setup runs on an embedded system from PC Engines. Hardware: APU6B4: 1 SFP, 3 LAN, quad core CPU, 4 GB RAM Modem: Simcom SIM7600G-H LTE, mini-pci Software OPNsense 24.7.3_1-amd64 FreeBSD 14.1-RELEASE-p3 OpenSSL 3.0.14 Here is my PPP log: 2024-09-03T00:22:06 Informational ppp [wan_link0] Link: reconnection attempt 8 in 2 seconds 2024-09-03T00:22:06 Informational ppp [wan_link0] LCP: Down event 2024-09-03T00:22:06 Informational ppp [wan_link0] Link: DOWN event 2024-09-03T00:22:06 Informational ppp [wan_link0] MODEM: chat script failed 2024-09-03T00:22:06 Informational ppp [wan_link0] CHAT: line 391: label "MomIdentGeneriicc" not found 2024-09-03T00:22:06 Informational ppp Label 'MomIdentGeneriicc' not found 2024-09-03T00:22:06 Informational ppp [wan_link0] Link: reconnection attempt 7
On the topic of the repeat strings in the AT command log (labeell and MomIdentGeneriicc): - expect or chat timing issues? Not unlike string buffers over serial interfaces with incorrect baud.
> https://cgit.freebsd.org/src/commit/?id=d3a83456e1e3 but that didn't help. One > thing of note in this commit's case is that it breaks the ascending order by > device ID. Please file a separate bug for that. The commit in question just added device IDs, but otherwise shouldn't have changed anything.
Looks like it's broken in 14.1 and good at some point in FreeBSD 13 - what's the latest known good revision? Are we sure that u3g is the right driver to look at? Could you please share dmesg output from the system in question, so we can see the device attach messages?
Here is my dmesg output from a the system (sim card currently unplugged) ``` ---<<BOOT>>--- Copyright (c) 1992-2023 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 14.1-RELEASE-p3 stable/24.7-n267802-bc5627fc5f3 SMP amd64 FreeBSD clang version 18.1.5 (https://github.com/llvm/llvm-project.git llvmorg-18.1.5-0-g617a15a9eac9) VT(vga): resolution 640x480 CPU: AMD GX-412TC SOC (998.16-MHz K8-class CPU) Origin="AuthenticAMD" Id=0x730f01 Family=0x16 Model=0x30 Stepping=1 Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT> Features2=0x3ed8220b<SSE3,PCLMULQDQ,MON,SSSE3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C> AMD Features=0x2e500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM> AMD Features2=0x1d4037ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,SKINIT,WDT,Topology,PNXC,DBE,PTSC,PL2I> Structured Extended Features=0x8<BMI1> XSAVE Features=0x1<XSAVEOPT> SVM: NP,NRIP,AFlush,DAssist,NAsids=8 TSC: P-state invariant, performance statistics real memory = 4294967296 (4096 MB) avail memory = 4084932608 (3895 MB) Event timer "LAPIC" quality 100 ACPI APIC Table: <COREv4 COREBOOT> FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs FreeBSD/SMP: 1 package(s) x 4 core(s) random: unblocking device. ioapic0 <Version 2.1> irqs 0-23 ioapic1 <Version 2.1> irqs 24-55 Launching APs: 2 1 3 random: entropy device external interface wlan: mac acl policy registered kbd0 at kbdmux0 WARNING: Device "spkr" is Giant locked and may be deleted before FreeBSD 15.0. vtvga0: <VT VGA driver> smbios0: <System Management BIOS> at iomem 0xf29e0-0xf29fe smbios0: Version: 3.0 aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS> acpi0: <COREv4 COREBOOT> acpi0: Power Button (fixed) atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0 atrtc0: registered as a time-of-day clock, resolution 1.000000s Event timer "RTC" frequency 32768 Hz quality 0 attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 apei0: <ACPI Platform Error Interface> on acpi0 hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 950 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <32-bit timer at 3.579545MHz> port 0x818-0x81b on acpi0 cpu0: <ACPI CPU> on acpi0 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 pci0: <ACPI PCI bus> on pcib0 pci0: <base peripheral, IOMMU> at device 0.2 (no driver attached) pcib1: <ACPI PCI-PCI bridge> irq 24 at device 2.1 on pci0 pcib1: failed to allocate initial I/O port window: 0x1000-0x1fff pci1: <ACPI PCI bus> on pcib1 igb0: <Intel(R) I210 (Fiber)> port 0x2000-0x201f mem 0xd0000000-0xd00fffff,0xd0200000-0xd0203fff irq 24 at device 0.0 on pci1 igb0: EEPROM V3.25-0 eTrack 0x800005d5 igb0: Using 1024 TX descriptors and 1024 RX descriptors igb0: Using 4 RX queues 4 TX queues igb0: Using MSI-X interrupts with 5 vectors igb0: Ethernet address: 00:0d:b9:63:2a:28 igb0: netmap queues/slots: TX 4/1024, RX 4/1024 pcib2: <ACPI PCI-PCI bridge> irq 25 at device 2.2 on pci0 pcib2: failed to allocate initial I/O port window: 0x2000-0x2fff pci2: <ACPI PCI bus> on pcib2 igb1: <Intel(R) I211 (Copper)> port 0x3000-0x301f mem 0xd0300000-0xd031ffff,0xd0320000-0xd0323fff irq 28 at device 0.0 on pci2 igb1: NVM V0.6 imgtype1 igb1: Using 1024 TX descriptors and 1024 RX descriptors igb1: Using 2 RX queues 2 TX queues igb1: Using MSI-X interrupts with 3 vectors igb1: Ethernet address: 00:0d:b9:63:2a:29 igb1: netmap queues/slots: TX 2/1024, RX 2/1024 pcib3: <ACPI PCI-PCI bridge> irq 26 at device 2.3 on pci0 pcib3: failed to allocate initial I/O port window: 0x3000-0x3fff pci3: <ACPI PCI bus> on pcib3 igb2: <Intel(R) I211 (Copper)> port 0x4000-0x401f mem 0xd0400000-0xd041ffff,0xd0420000-0xd0423fff irq 32 at device 0.0 on pci3 igb2: NVM V0.6 imgtype1 igb2: Using 1024 TX descriptors and 1024 RX descriptors igb2: Using 2 RX queues 2 TX queues igb2: Using MSI-X interrupts with 3 vectors igb2: Ethernet address: 00:0d:b9:63:2a:2a igb2: netmap queues/slots: TX 2/1024, RX 2/1024 pcib4: <ACPI PCI-PCI bridge> irq 27 at device 2.4 on pci0 pcib4: failed to allocate initial I/O port window: 0x4000-0x4fff pci4: <ACPI PCI bus> on pcib4 igb3: <Intel(R) I211 (Copper)> port 0x6000-0x601f mem 0xd0500000-0xd051ffff,0xd0520000-0xd0523fff irq 36 at device 0.0 on pci4 igb3: NVM V0.6 imgtype1 igb3: Using 1024 TX descriptors and 1024 RX descriptors igb3: Using 2 RX queues 2 TX queues igb3: Using MSI-X interrupts with 3 vectors igb3: Ethernet address: 00:0d:b9:63:2a:2b igb3: netmap queues/slots: TX 2/1024, RX 2/1024 pci0: <encrypt/decrypt> at device 8.0 (no driver attached) xhci0: <AMD FCH USB 3.0 controller> mem 0xd0822000-0xd0823fff irq 18 at device 16.0 on pci0 xhci0: 32 bytes context size, 64-bit DMA usbus0 on xhci0 usbus0: 5.0Gbps Super Speed USB v3.0 ahci0: <AMD Hudson-2 AHCI SATA controller> port 0x5010-0x5017,0x5020-0x5023,0x5018-0x501f,0x5024-0x5027,0x5000-0x500f mem 0xd0825000-0xd08253ff at device 17.0 on pci0 ahci0: AHCI v1.30 with 2 6Gbps ports, Port Multiplier supported with FBS ahcich0: <AHCI channel> at channel 0 on ahci0 ahcich1: <AHCI channel> at channel 1 on ahci0 ehci0: <AMD FCH USB 2.0 controller> mem 0xd0826000-0xd08260ff irq 18 at device 18.0 on pci0 usbus1: EHCI version 1.0 usbus1 on ehci0 usbus1: 480Mbps High Speed USB v2.0 ehci1: <AMD FCH USB 2.0 controller> mem 0xd0827000-0xd08270ff irq 18 at device 19.0 on pci0 usbus2: EHCI version 1.0 usbus2 on ehci1 usbus2: 480Mbps High Speed USB v2.0 isab0: <PCI-ISA bridge> at device 20.3 on pci0 isa0: <ISA bus> on isab0 sdhci_pci0: <Generic SD HCI> mem 0xd0828000-0xd08280ff at device 20.7 on pci0 sdhci_pci0: 1 slot(s) allocated acpi_tz0: <Thermal Zone> on acpi0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: console (115200,n,8,1) uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 orm0: <ISA Option ROM> at iomem 0xee800-0xeffff pnpid ORM0000 on isa0 hwpstate0: <Cool`n'Quiet 2.0> on cpu0 Timecounter "TSC" frequency 998128185 Hz quality 1000 Timecounters tick every 1.000 msec ugen0.1: <AMD XHCI root HUB> at usbus0 ugen2.1: <AMD EHCI root HUB> at usbus2 ugen1.1: <AMD EHCI root HUB> at usbus1 uhub0 on usbus0 uhub0: <AMD XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0 uhub1 on usbus1 uhub1: <AMD EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1 uhub2 on usbus2 uhub2: <AMD EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus2 ZFS filesystem version: 5 ZFS storage pool version: features support (5000) ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 ada0: <YSE128GTLCW-SBC-2 SBFM61.5> ACS-4 ATA SATA 3.x device ada0: Serial Number 511230621109001052 ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 122104MB (250069680 512 byte sectors) Trying to mount root from zfs:zroot/ROOT/default []... uhub0: 4 ports with 4 removable, self powered uhub1: 2 ports with 2 removable, self powered uhub2: 2 ports with 2 removable, self powered Root mount waiting for: usbus1 usbus2 ugen1.2: <vendor 0x0438 product 0x7900> at usbus1 uhub3 on uhub1 uhub3: <vendor 0x0438 product 0x7900, class 9/0, rev 2.00/0.18, addr 2> on usbus1 ugen2.2: <vendor 0x0438 product 0x7900> at usbus2 uhub4 on uhub2 uhub4: <vendor 0x0438 product 0x7900, class 9/0, rev 2.00/0.18, addr 2> on usbus2 uhub3: 4 ports with 4 removable, self powered uhub4: 4 ports with 4 removable, self powered ugen2.3: <SimTech, Incorporated SimTech, Incorporated> at usbus2 u3g0 on uhub4 u3g0: <SimTech, Incorporated SimTech, Incorporated, class 0/0, rev 2.00/3.18, addr 3> on usbus2 u3g0: Found 6 ports. igb0: link state changed to UP igb1: link state changed to UP igb2: link state changed to UP intsmb0: <AMD FCH SMBus Controller> at device 20.0 on pci0 smbus0: <System Management Bus> on intsmb0 lo0: link state changed to UP pflog0: permanently promiscuous mode enabled igb2: link state changed to DOWN wg0: link state changed to UP igb1: link state changed to DOWN igb0: link state changed to DOWN igb0: link state changed to UP igb2: link state changed to UP igb1: link state changed to UP ```
(In reply to Warner Losh from comment #4) https://github.com/freebsd/freebsd-src/pull/1410 Could this have something to do with changing the default serial bps in FreeBSD 14? It's quite odd that it only affects a tiny portion of modems. https://cgit.freebsd.org/src/commit/?id=4722ceb7d53e Cheers, Franco
No, so what seems to be the case is that mpd5 is misbehaving on 14.1 vs. 13.2 for whatever reason: > set $ErrorMsg "The modem is not responding to \"AT$modemCmd\" at ModemCmd: label." What mpd5 says: > CHAT: The modem is not responding to "AT" at MomCmd: labeell. You can clearly see that it mangled "ModemCmd" into "MomCmd" and adding the weird trailing characters into the sentence "eell". One could argue the mpd.script file is mangled, but it doesn't make any sense since the file is rewritten on each connection invoke. The other one is: "MomIdentGeneriicc" Which is supposed to be: "ModemIdentGeneric" So probably the mpd5 source code, possibly inside ChatExpandString() that reads labels and other things from the mpd.script file. I've added debug there and we'll see what happens. Cheers, Franco
^Triage: note that this is a possible regression in mpd5. However, it's not clear to me if this is src mpd5 or ports mpd5?
net/mpd5, but it's only a theory. It's rather difficult to reproduce it seems.
For what it's worth...Mpd5 was working for me on 14.0. Just upgraded to 14.1 and upgraded packages (Poudriere) at the same time, now my Sierra Wireless modem is not working. Inspecting the log shows the same issues Franco reported: [wan_link0] CHAT: matched set "", goto label "MomIdentGeneriicc" Label 'MomIdentGeneriicc' not found IF the issue is with Mpd5, it is likely a regression in the FreeBSD Mpd5 port, as it was updated relatively recently and source wasn't...but that's just a guess.
The mpd5 port didn't change in that regard as far as I can tell, so it is some sort of interaction with the build system if 14.0 indeed worked fine (in OPnsense we can only conclude that 13.2 worked fine because we never used 14.0) and now 14.1 doesn't. It could also be a runtime issue. A test with the binary of 14.0 on 14.1 might make this clearer if it would run. Cheers, Franco
Upgraded to 14.0 on 2-16, have used mpd5 daily since with no issue. Just upgraded to 14.1, and it no longer works. Reverted to PPP until resolved.
Did some testing, and it seems mpd5 is not the issue. The folks with FreeBSD logos in their name seem to know that, just trying to help narrow the focus. Setting pkg to use https://pkg.freebsd.org/FreeBSD:14:amd64/release_0/ pulls in mpd5 from October, and it produces the same behavior. mpd5.log - FreeBSD 14.0 - Feb 25 2024 until Oct 1 2024 [wan_link0] CHAT: line 356: match "OK^M " ModemIdentGeneric [wan_link0] CHAT: line 357: wait 3 mpd5.log - FreeBSD 14.1 - Oct 2 2024 (regardless of mpd5 build date) [wan_link0] CHAT: line 356: match "OK^M " MomIdentGeneriicc [wan_link0] CHAT: line 357: label "MomIdentGeneriicc" not found Tested on similar hardware to what is show in comment #6
Also, using a Sierra modem via USB, in case that helps narrow (or expand) the search. ugen0.2: <Sierra Wireless, Incorporated Sierra Wireless MC7455 Qualcomm Snapdragon X7 LTE-A> at usbus0 u3g0 on uhub2 u3g0: <Sierra Wireless, Incorporated Sierra Wireless MC7455 Qualcomm Snapdragon X7 LTE-A, class 0/0, rev 2.10/0.06, addr 1> on usbus0 u3g0: Found 5 ports.
I confirm the same problems as in this comment #1, on a Fibocom L850-GL modem (assuming it's an Intel XMM7360), I'm using a USB adapter. I see that a user in 2022 (https://forum.opnsense.org/index.php?topic=26289.msg149573#msg149573) was using a similar model (firmware structure is not much different), he had a PPP connection working fine, supporting the theory that the problem occurred after the FreeBSD upgrade. I can help with testing or providing logs, but unfortunately I'm not an advanced user, so I can't help with source code or anything like that. Some info: OPNsense 24.7.5_3-amd64 FreeBSD 14.1-RELEASE-p5 OpenSSL 3.0.15 Product link: https://aliexpress.com/item/4001066330406.html Modem works fine on Windows in MBIM and NCM modes (with script).
I'm also running into the same issue using a Sierra EM7455 (Dell DW5811e flashed to generic EM7455) with OPNsense 24.7 (FreeBSD 14.1). Re-installing OPNsense 24.1 (FreeBSD 13.2-RELEASE-p11) resolved the issue for me.
Same problem here and I have an external LTE modem so I will use it until the problem is fixed. If you need any testing i´m available.