Bug 244932

Summary: [patch] mail/dovecot: use libexttextcat-2.0 instead of libtextcat, working Clucene with it
Product: Ports & Packages Reporter: Igor Zabelin <igorz>
Component: Individual Port(s)Assignee: Larry Rosenman <ler>
Status: Closed FIXED    
Severity: Affects Many People CC: lysfjord.daniel
Priority: --- Flags: bugzilla: maintainer-feedback? (ler)
Version: Latest   
Hardware: Any   
OS: Any   
Description Flags
patch-dovecot-libexttextcat-clucene none

Description Igor Zabelin 2020-03-20 09:27:23 UTC
Created attachment 212546 [details]


2015-04-20 18:08:56 +0300 Timo Sirainen <tss@iki.fi> (82eadbc43)

    lib-fts requires libexttextcat actually - don't even try to use textcat for
    it. I'm also not sure yet if some libexttextcats are too old for it.
Comment 1 Igor Zabelin 2020-03-23 21:10:35 UTC
Created attachment 212656 [details]

At now CLucente is broken with textсat:

Fatal: Couldn't load required plugin /usr/local/lib/dovecot/lib21_fts_lucene_plugin.so: dlopen() failed: /usr/local/lib/dovecot/lib21_fts_lucene_plugin.so: Undefined symbol "_ZN6lucene8analysis8snowball16SnowballAnalyzerC1EPFiPKvmP6bufferEPKc"

Libstemmer is missing but we can use it from CLucene.

Comment 2 commit-hook freebsd_committer 2020-03-23 22:08:31 UTC
A commit references this bug:

Author: ler
Date: Mon Mar 23 22:07:59 UTC 2020
New revision: 529013
URL: https://svnweb.freebsd.org/changeset/ports/529013

  mail/dovecot: use libexttextcat for lucene.

  PR:		244932
  Submitted by:	igorz@yandex.ru

Comment 3 Larry Rosenman freebsd_committer 2020-03-23 22:09:54 UTC
Committed, thanks!
Comment 4 lysfjord.daniel 2020-03-26 13:48:55 UTC
After this patch, any attempts to index anything causes this error. Using clucene.

Error: lucene index /var/mail/redacted.maildir/lucene-indexes: IndexWriter::addDocument() failed (#4): language not available for stemming
Comment 5 Igor Zabelin 2020-03-26 14:09:45 UTC
There is no clear description of the resolved problem.
Before patch, if you compiled textcat and fts-lucene did't work at all. 
Now you can use textcat for fts, but probably not for fts-lucene.

In any case, now lucene is a fast simple useful search.


Timo Sirainen tss at iki.fi
Wed Jun 17 13:43:17 UTC 2015


The textcat support in fts-lucene works very badly and shouldn't be used. There's new lib-fts code being developed that supports multiple languages better. It's already kind of usable in v2.2.18, but would be better to wait for v2.2.19.
Comment 6 Igor Zabelin 2020-03-26 14:27:26 UTC
My config:

plugin {
        plugin = fts fts_lucene

        fts = lucene
        fts_lucene = whitespace_chars=@. no_snowball normalize

        fts_autoindex = yes
        fts_enforced = yes

You can also try add

        fts_filters = normalizer-icu snowball

or only snowball.

What do you have in textcat_conf=<path> text cat_dir=<path> ?
You can try making the correct textcat_conf for the content of /usr/local/share/libexttextcat

is not correct.
Comment 7 lysfjord.daniel 2020-03-26 20:53:52 UTC
(In reply to Igor Zabelin from comment #6)

fts_lucene = whitespace_chars=@. no_snowball normalize

The above worked. Thanks:)
Comment 8 Igor Zabelin 2020-03-26 21:12:51 UTC
Here about fts_filters:


Timo Sirainen tss at iki.fi
Thu Jun 30 13:11:05 UTC 2016

Yet another possibility could be to use Dovecot's new lib-fts, which can also be used with fts-lucene. It's still basically the same snowball+textcat though, but made to work better. But looks like it's not documented yet in wiki..
Comment 9 lysfjord.daniel 2020-03-26 21:33:32 UTC
(In reply to Igor Zabelin from comment #6)
Note, I had nothing in neither textcat_conf=<path>, nor text cat_dir=<path>, never had.
Comment 10 commit-hook freebsd_committer 2020-05-18 20:50:28 UTC
A commit references this bug:

Author: ler
Date: Mon May 18 20:50:14 UTC 2020
New revision: 535786
URL: https://svnweb.freebsd.org/changeset/ports/535786

  MFH: r529013 r535778

  mail/dovecot: use libexttextcat for lucene.

  PR:		244932
  Submitted by:	igorz@yandex.ru

  mail/dovecot: Upgrade to, fixing multiple vulnerabilities.

  - CVE-2020-10957: lmtp/submission: A client can crash the server by
    sending a NOOP command with an invalid string parameter. This occurs
    particularly for a parameter that doesn't start with a double quote.
    This applies to all SMTP services, including submission-login, which
    makes it possible to crash the submission service without
  - CVE-2020-10958: lmtp/submission: Sending many invalid or unknown
    commands can cause the server to access freed memory, which can lead
    to a server crash. This happens when the server closes the connection
    with a "421 Too many invalid commands" error. The bad command limit
    depends on the service (lmtp or submission) and varies between 10 to
    20 bad commands.
  - CVE-2020-10967: lmtp/submission: Issuing the RCPT command with an
    address that has the empty quoted string as local-part causes the
    lmtp service to crash.

  Clean up some REINPLACE warnings whilst we're here.

  Security:	37d106a8-15a4-483e-8247-fcb68b16eaf8
  Security:	CVE-2020-10957
  Security:	CVE-2020-10958
  Security:	CVE-2020-10967

  Approved by:	ports-secteam (joneum)

_U  branches/2020Q2/