Bug 244932 - [patch] mail/dovecot: use libexttextcat-2.0 instead of libtextcat, working Clucene with it
Summary: [patch] mail/dovecot: use libexttextcat-2.0 instead of libtextcat, working Cl...
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Many People
Assignee: Larry Rosenman
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-03-20 09:27 UTC by Igor Zabelin
Modified: 2020-05-18 20:50 UTC (History)
1 user (show)

See Also:
bugzilla: maintainer-feedback? (ler)


Attachments
patch-dovecot-libexttextcat (695 bytes, patch)
2020-03-20 09:27 UTC, Igor Zabelin
no flags Details | Diff
patch-dovecot-libexttextcat-clucene (4.17 KB, patch)
2020-03-23 21:10 UTC, Igor Zabelin
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Igor Zabelin 2020-03-20 09:27:23 UTC
Created attachment 212546 [details]
patch-dovecot-libexttextcat

dovecot-2.3.10/ChangeLog:

2015-04-20 18:08:56 +0300 Timo Sirainen <tss@iki.fi> (82eadbc43)

    lib-fts requires libexttextcat actually - don't even try to use textcat for
    it. I'm also not sure yet if some libexttextcats are too old for it.
Comment 1 Igor Zabelin 2020-03-23 21:10:35 UTC
Created attachment 212656 [details]
patch-dovecot-libexttextcat-clucene

At now CLucente is broken with textсat:

Fatal: Couldn't load required plugin /usr/local/lib/dovecot/lib21_fts_lucene_plugin.so: dlopen() failed: /usr/local/lib/dovecot/lib21_fts_lucene_plugin.so: Undefined symbol "_ZN6lucene8analysis8snowball16SnowballAnalyzerC1EPFiPKvmP6bufferEPKc"

Libstemmer is missing but we can use it from CLucene.

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=175813
Comment 2 commit-hook freebsd_committer 2020-03-23 22:08:31 UTC
A commit references this bug:

Author: ler
Date: Mon Mar 23 22:07:59 UTC 2020
New revision: 529013
URL: https://svnweb.freebsd.org/changeset/ports/529013

Log:
  mail/dovecot: use libexttextcat for lucene.

  PR:		244932
  Submitted by:	igorz@yandex.ru

Changes:
  head/mail/dovecot/Makefile
  head/mail/dovecot/files/patch-configure
  head/mail/dovecot/files/patch-plugins_fts-lucene_Makefile.in
  head/mail/dovecot/files/patch-src_lib-fts_Makefile.in
  head/mail/dovecot/files/patch-src_lib-fts_fts-filter-stemmer-snowball.c
  head/mail/dovecot/files/patch-src_plugins_fts-lucene_SnowballFilter.h
Comment 3 Larry Rosenman freebsd_committer 2020-03-23 22:09:54 UTC
Committed, thanks!
Comment 4 lysfjord.daniel 2020-03-26 13:48:55 UTC
After this patch, any attempts to index anything causes this error. Using clucene.

Error: lucene index /var/mail/redacted.maildir/lucene-indexes: IndexWriter::addDocument() failed (#4): language not available for stemming
Comment 5 Igor Zabelin 2020-03-26 14:09:45 UTC
There is no clear description of the resolved problem.
Before patch, if you compiled textcat and fts-lucene did't work at all. 
Now you can use textcat for fts, but probably not for fts-lucene.

In any case, now lucene is a fast simple useful search.

https://dovecot.org/list/dovecot/2015-June/101205.html

Timo Sirainen tss at iki.fi
Wed Jun 17 13:43:17 UTC 2015

[..]

The textcat support in fts-lucene works very badly and shouldn't be used. There's new lib-fts code being developed that supports multiple languages better. It's already kind of usable in v2.2.18, but would be better to wait for v2.2.19.
Comment 6 Igor Zabelin 2020-03-26 14:27:26 UTC
My config:

plugin {
        plugin = fts fts_lucene

        fts = lucene
        fts_lucene = whitespace_chars=@. no_snowball normalize

        fts_autoindex = yes
        fts_enforced = yes
}

You can also try add

        fts_filters = normalizer-icu snowball

or only snowball.

What do you have in textcat_conf=<path> text cat_dir=<path> ?
You can try making the correct textcat_conf for the content of /usr/local/share/libexttextcat

./work/dovecot-2.3.10/src/plugins/fts-lucene/textcat.conf
is not correct.
Comment 7 lysfjord.daniel 2020-03-26 20:53:52 UTC
(In reply to Igor Zabelin from comment #6)

fts_lucene = whitespace_chars=@. no_snowball normalize

The above worked. Thanks:)
Comment 8 Igor Zabelin 2020-03-26 21:12:51 UTC
Here about fts_filters:

https://dovecot.org/pipermail/dovecot/2016-June/104772.html

Timo Sirainen tss at iki.fi
Thu Jun 30 13:11:05 UTC 2016

[...]
Yet another possibility could be to use Dovecot's new lib-fts, which can also be used with fts-lucene. It's still basically the same snowball+textcat though, but made to work better. But looks like it's not documented yet in wiki..
Comment 9 lysfjord.daniel 2020-03-26 21:33:32 UTC
(In reply to Igor Zabelin from comment #6)
Note, I had nothing in neither textcat_conf=<path>, nor text cat_dir=<path>, never had.
Comment 10 commit-hook freebsd_committer 2020-05-18 20:50:28 UTC
A commit references this bug:

Author: ler
Date: Mon May 18 20:50:14 UTC 2020
New revision: 535786
URL: https://svnweb.freebsd.org/changeset/ports/535786

Log:
  MFH: r529013 r535778

  mail/dovecot: use libexttextcat for lucene.

  PR:		244932
  Submitted by:	igorz@yandex.ru

  mail/dovecot: Upgrade to 2.3.10.1, fixing multiple vulnerabilities.

  - CVE-2020-10957: lmtp/submission: A client can crash the server by
    sending a NOOP command with an invalid string parameter. This occurs
    particularly for a parameter that doesn't start with a double quote.
    This applies to all SMTP services, including submission-login, which
    makes it possible to crash the submission service without
    authentication.
  - CVE-2020-10958: lmtp/submission: Sending many invalid or unknown
    commands can cause the server to access freed memory, which can lead
    to a server crash. This happens when the server closes the connection
    with a "421 Too many invalid commands" error. The bad command limit
    depends on the service (lmtp or submission) and varies between 10 to
    20 bad commands.
  - CVE-2020-10967: lmtp/submission: Issuing the RCPT command with an
    address that has the empty quoted string as local-part causes the
    lmtp service to crash.

  Clean up some REINPLACE warnings whilst we're here.

  Security:	37d106a8-15a4-483e-8247-fcb68b16eaf8
  Security:	CVE-2020-10957
  Security:	CVE-2020-10958
  Security:	CVE-2020-10967

  Approved by:	ports-secteam (joneum)

Changes:
_U  branches/2020Q2/
  branches/2020Q2/mail/dovecot/Makefile
  branches/2020Q2/mail/dovecot/distinfo