Bug 244932

Created attachment 212656 [details]
patch-dovecot-libexttextcat-clucene

At now CLucente is broken with textсat:

Fatal: Couldn't load required plugin /usr/local/lib/dovecot/lib21_fts_lucene_plugin.so: dlopen() failed: /usr/local/lib/dovecot/lib21_fts_lucene_plugin.so: Undefined symbol "_ZN6lucene8analysis8snowball16SnowballAnalyzerC1EPFiPKvmP6bufferEPKc"

Libstemmer is missing but we can use it from CLucene.

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=175813

Comment 2 commit-hook 2020-03-23 22:08:31 UTC

A commit references this bug:

Author: ler
Date: Mon Mar 23 22:07:59 UTC 2020
New revision: 529013
URL: https://svnweb.freebsd.org/changeset/ports/529013

Log:
  mail/dovecot: use libexttextcat for lucene.

  PR:		244932
  Submitted by:	igorz@yandex.ru

Changes:
  head/mail/dovecot/Makefile
  head/mail/dovecot/files/patch-configure
  head/mail/dovecot/files/patch-plugins_fts-lucene_Makefile.in
  head/mail/dovecot/files/patch-src_lib-fts_Makefile.in
  head/mail/dovecot/files/patch-src_lib-fts_fts-filter-stemmer-snowball.c
  head/mail/dovecot/files/patch-src_plugins_fts-lucene_SnowballFilter.h

Comment 3 Larry Rosenman 2020-03-23 22:09:54 UTC

Committed, thanks!

After this patch, any attempts to index anything causes this error. Using clucene.

Error: lucene index /var/mail/redacted.maildir/lucene-indexes: IndexWriter::addDocument() failed (#4): language not available for stemming

There is no clear description of the resolved problem.
Before patch, if you compiled textcat and fts-lucene did't work at all. 
Now you can use textcat for fts, but probably not for fts-lucene.

In any case, now lucene is a fast simple useful search.

https://dovecot.org/list/dovecot/2015-June/101205.html

Timo Sirainen tss at iki.fi
Wed Jun 17 13:43:17 UTC 2015

[..]

The textcat support in fts-lucene works very badly and shouldn't be used. There's new lib-fts code being developed that supports multiple languages better. It's already kind of usable in v2.2.18, but would be better to wait for v2.2.19.

My config:

plugin {
        plugin = fts fts_lucene

        fts = lucene
        fts_lucene = whitespace_chars=@. no_snowball normalize

        fts_autoindex = yes
        fts_enforced = yes
}

You can also try add

        fts_filters = normalizer-icu snowball

or only snowball.

What do you have in textcat_conf=<path> text cat_dir=<path> ?
You can try making the correct textcat_conf for the content of /usr/local/share/libexttextcat

./work/dovecot-2.3.10/src/plugins/fts-lucene/textcat.conf
is not correct.

(In reply to Igor Zabelin from comment #6)

fts_lucene = whitespace_chars=@. no_snowball normalize

The above worked. Thanks:)

Here about fts_filters:

https://dovecot.org/pipermail/dovecot/2016-June/104772.html

Timo Sirainen tss at iki.fi
Thu Jun 30 13:11:05 UTC 2016

[...]
Yet another possibility could be to use Dovecot's new lib-fts, which can also be used with fts-lucene. It's still basically the same snowball+textcat though, but made to work better. But looks like it's not documented yet in wiki..

(In reply to Igor Zabelin from comment #6)
Note, I had nothing in neither textcat_conf=<path>, nor text cat_dir=<path>, never had.

Comment 10 commit-hook 2020-05-18 20:50:28 UTC

A commit references this bug:

Author: ler
Date: Mon May 18 20:50:14 UTC 2020
New revision: 535786
URL: https://svnweb.freebsd.org/changeset/ports/535786

Log:
  MFH: r529013 r535778

  mail/dovecot: use libexttextcat for lucene.

  PR:		244932
  Submitted by:	igorz@yandex.ru

  mail/dovecot: Upgrade to 2.3.10.1, fixing multiple vulnerabilities.

  - CVE-2020-10957: lmtp/submission: A client can crash the server by
    sending a NOOP command with an invalid string parameter. This occurs
    particularly for a parameter that doesn't start with a double quote.
    This applies to all SMTP services, including submission-login, which
    makes it possible to crash the submission service without
    authentication.
  - CVE-2020-10958: lmtp/submission: Sending many invalid or unknown
    commands can cause the server to access freed memory, which can lead
    to a server crash. This happens when the server closes the connection
    with a "421 Too many invalid commands" error. The bad command limit
    depends on the service (lmtp or submission) and varies between 10 to
    20 bad commands.
  - CVE-2020-10967: lmtp/submission: Issuing the RCPT command with an
    address that has the empty quoted string as local-part causes the
    lmtp service to crash.

  Clean up some REINPLACE warnings whilst we're here.

  Security:	37d106a8-15a4-483e-8247-fcb68b16eaf8
  Security:	CVE-2020-10957
  Security:	CVE-2020-10958
  Security:	CVE-2020-10967

  Approved by:	ports-secteam (joneum)

Changes:
_U  branches/2020Q2/
  branches/2020Q2/mail/dovecot/Makefile
  branches/2020Q2/mail/dovecot/distinfo