Bug 266629 - contrib/dma corrupted queue files
Summary: contrib/dma corrupted queue files
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: bin (show other bugs)
Version: 13.1-STABLE
Hardware: Any Any
: --- Affects Only Me
Assignee: Ed Maste
URL: https://reviews.freebsd.org/D36763
Keywords:
Depends on:
Blocks:
 
Reported: 2022-09-26 13:03 UTC by Mikko Lehto
Modified: 2022-11-18 21:00 UTC (History)
1 user (show)

See Also:


Attachments
Helper script for highlighting failing local submission (883 bytes, text/plain)
2022-09-26 21:34 UTC, Mikko Lehto
no flags Details
Sample log (1.19 KB, text/plain)
2022-09-26 21:51 UTC, Mikko Lehto
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Mikko Lehto 2022-09-26 13:03:40 UTC
DragonFly Mail Agent DMA from base system (/usr/libexec/dma, version ~0.11)  sometimes reports corrupted queue files.

When I switch to DMA from binary pkg (/usr/local/libexec/dma, version ~0.13) then errors regarding corrupted queue files are not occuring anymore.

Slightly more verbose report in upstream issue tracker:
  https://github.com/corecode/dma/issues/110
Comment 1 Mikko Lehto 2022-09-26 13:29:14 UTC
Looks like FreeBSD main branch has this additional commit that is not in FreeBSD 13.1.

https://cgit.freebsd.org/src/commit/contrib/dma/mail.c?id=d21e71efce3963b068ccaa807a099b591dc220e9

https://reviews.freebsd.org/D34159
Comment 2 Ed Maste freebsd_committer freebsd_triage 2022-09-26 14:29:32 UTC
(In reply to Mikko Lehto from comment #1)
I just now merged that commit to stable/13 in 2e8403e0215868a13e2bdd1c979fc351971b8d91
Comment 3 Mikko Lehto 2022-09-26 18:35:49 UTC
(In reply to Ed Maste from comment #2)

Thank you Ed, unfortunately still gives "corrupted queue file".
Hope this helps, looking forward not to use Sendmail.


Below are steps to reproduce:

1. Succesful case with no problems (13.1 binary pkg mail/dma-0.13.20220128_1,1):
---
echo -n 'hello world' | /usr/local/libexec/dma -bq -f localuser@localhost remouser@remotehost.example.invalid
---
In this OK case queue file /var/spool/dma/M... ends in byte 0x0A, that seem to be the expected queue file body suffix.
Remote delivery is succesful.

                                       
2. Failing case (13.1 src.txz + new mail.c from 2e8403e0215868a13e2bdd1c979fc351971b8d91):
---
echo -n 'hello world' | /usr/libexec/dma_266629 -bq -f localuser@localhost remouser@remotehost.example.invalid
---
In this failing case I can see that queue file /var/spool/dma/M... ends in 0x64, that is last letter of "world".
Remote delivery fails.


I use "hexdump -C /var/spool/dma/M..." to observe last byte in queue file.


For fun I also took mail.c from upstream repository and compiled.
With this, frankenstein, there is always 0x0A at the end of /var/spool/dma/M... queue file.


( Still comparing readmail() loops, trying to figure something )
Comment 4 Mikko Lehto 2022-09-26 21:34:42 UTC
Created attachment 236845 [details]
Helper script for highlighting failing local submission

Attached helper script "test_dma.sh" that aims to highlight problematic local submission scenario.
Comment 5 Mikko Lehto 2022-09-26 21:51:43 UTC
Created attachment 236848 [details]
Sample log

Sample output with test_dma.sh script.

This run highlights queue identifiers:
5ff8e.801c48000
and 
5ff96.800e48000

They are scenarios:
"/usr/libexec/dma BODY WITHOUT NEWLINE"
and
"/usr/libexec/dma_266629 BODY WITHOUT NEWLINE"
Comment 6 Ed Maste freebsd_committer freebsd_triage 2022-09-27 14:36:29 UTC
OK to be sure I understand correctly, the issue is that when submitting mail where the body does not end with a newline, upstream dma adds a newline to the queued mail while the version in the base system does not?

The version in the base system has an additional change to split long lines - see https://github.com/corecode/dma/issues/18 - and presumably this is not adding the newline when the input does not have one.
Comment 7 Mikko Lehto 2022-09-28 15:22:51 UTC
(In reply to Ed Maste from comment #6)

Yes, you just described it the way I also understand this issue, thanks.

At first I was uncertain whether "Sendmail pipe interface" expects some special
sequence of characters that might cause this, but I think my calling style is
expected.

Your patch[1] seem to work, I can not reproduce "corrupted queue files" any more, great.

[1] https://reviews.freebsd.org/D36763
Comment 8 commit-hook freebsd_committer freebsd_triage 2022-10-12 16:00:29 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=169487223143b1232ec4686b720b028af8d6d42b

commit 169487223143b1232ec4686b720b028af8d6d42b
Author:     Ed Maste <emaste@FreeBSD.org>
AuthorDate: 2022-09-27 19:19:19 +0000
Commit:     Ed Maste <emaste@FreeBSD.org>
CommitDate: 2022-10-12 15:59:01 +0000

    dma: restore addition of newline when missing from input

    If input mail does not have a newline on the last line dma must add
    one.  This was broken by the addition of long-line splitting, with the
    switch from strlen(line) to linelen returned by getline().

    PR:             266629
    Reviewed by:    bapt, Mikko Lehto
    Tested by:      Mikko Lehto
    MFC after:      1 week
    Fixes:          b0b2d05fd060 ("Split body of mails not respecting...")
    Sponsored by:   The FreeBSD Foundation
    Differential Revision:  https://reviews.freebsd.org/D36763

 contrib/dma/mail.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
Comment 9 commit-hook freebsd_committer freebsd_triage 2022-10-19 01:04:57 UTC
A commit in branch stable/13 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=bd3d597a6bc0e35273669cfd7a99b3a5b437d92a

commit bd3d597a6bc0e35273669cfd7a99b3a5b437d92a
Author:     Ed Maste <emaste@FreeBSD.org>
AuthorDate: 2022-09-27 19:19:19 +0000
Commit:     Ed Maste <emaste@FreeBSD.org>
CommitDate: 2022-10-19 01:03:55 +0000

    dma: restore addition of newline when missing from input

    If input mail does not have a newline on the last line dma must add
    one.  This was broken by the addition of long-line splitting, with the
    switch from strlen(line) to linelen returned by getline().

    PR:             266629
    Reviewed by:    bapt, Mikko Lehto
    Tested by:      Mikko Lehto
    MFC after:      1 week
    Fixes:          b0b2d05fd060 ("Split body of mails not respecting...")
    Sponsored by:   The FreeBSD Foundation
    Differential Revision:  https://reviews.freebsd.org/D36763

    (cherry picked from commit 169487223143b1232ec4686b720b028af8d6d42b)

 contrib/dma/mail.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
Comment 10 commit-hook freebsd_committer freebsd_triage 2022-11-18 21:00:46 UTC
A commit in branch stable/12 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=de0b782545f725e719f9b3f1ec7f2e6442551086

commit de0b782545f725e719f9b3f1ec7f2e6442551086
Author:     Ed Maste <emaste@FreeBSD.org>
AuthorDate: 2022-09-27 19:19:19 +0000
Commit:     Ed Maste <emaste@FreeBSD.org>
CommitDate: 2022-11-18 20:57:45 +0000

    dma: restore addition of newline when missing from input

    If input mail does not have a newline on the last line dma must add
    one.  This was broken by the addition of long-line splitting, with the
    switch from strlen(line) to linelen returned by getline().

    PR:             266629
    Reviewed by:    bapt, Mikko Lehto
    Tested by:      Mikko Lehto
    MFC after:      1 week
    Fixes:          b0b2d05fd060 ("Split body of mails not respecting...")
    Sponsored by:   The FreeBSD Foundation
    Differential Revision:  https://reviews.freebsd.org/D36763

    (cherry picked from commit 169487223143b1232ec4686b720b028af8d6d42b)
    (cherry picked from commit bd3d597a6bc0e35273669cfd7a99b3a5b437d92a)

 contrib/dma/mail.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)