Bug 257728 - databases/mariadb105-server: 10.5.12 causes corrupted TRX_NO 488005401dea0a2
Summary: databases/mariadb105-server: 10.5.12 causes corrupted TRX_NO 488005401dea0a2
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Bernard Spil
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2021-08-10 07:17 UTC by Matthias Fechner
Modified: 2021-10-28 12:43 UTC (History)
7 users (show)

See Also:
bugzilla: maintainer-feedback? (brnrd)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Matthias Fechner freebsd_committer freebsd_triage 2021-08-10 07:17:45 UTC
I rolled this now version now out, but one server get a direct crash on start:
2021-08-10  8:53:26 0 [Note] InnoDB: Uses event mutexes
2021-08-10  8:53:26 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2021-08-10  8:53:26 0 [Note] InnoDB: Number of pools: 1
2021-08-10  8:53:26 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2021-08-10  8:53:26 0 [Note] InnoDB: Initializing buffer pool, total size = 1073741824, chunk size = 134217728
2021-08-10  8:53:27 0 [Note] InnoDB: Completed initialization of buffer pool
2021-08-10  8:53:27 0 [ERROR] InnoDB: corrupted TRX_NO 488005401dea0a2
2021-08-10  8:53:27 0 [Note] InnoDB: Retry with innodb_force_recovery=5
2021-08-10  8:53:27 0 [ERROR] InnoDB: Plugin initialization aborted with error Data structure corruption
2021-08-10  8:53:27 0 [Note] InnoDB: Starting shutdown...
2021-08-10  8:53:27 0 [ERROR] Plugin 'InnoDB' init function returned error.
2021-08-10  8:53:27 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2021-08-10  8:53:27 0 [Note] Plugin 'FEEDBACK' is disabled.
2021-08-10  8:53:27 0 [ERROR] Unknown/unsupported storage engine: InnoDB
2021-08-10  8:53:27 0 [ERROR] Aborting

I tried it with different database backup, all the same problem.
After a downgrade to 10.5.11 it was working again.

This FreeBSD server is running on a virtual platform (I think it is KVM but I'm not sure).
I'm not sure if this crash is related that FreeBSD runs as a virtual machine, but as all my other machines are running fine with the new version, it could be related to this.
Comment 1 Matthias Fechner freebsd_committer freebsd_triage 2021-08-10 07:19:54 UTC
I think the build logs are not important, but just to be complete.

The build log for 10.5.11:
https://pkg.fechner.net/data/130amd64-default/2021-08-10_08h57m30s/logs/mariadb105-server-10.5.11.log

The build log for 10.5.12:
https://pkg.fechner.net/data/130amd64-default/2021-08-09_10h05m12s/logs/mariadb105-server-10.5.12.log
Comment 2 Nathan Robertson 2021-08-13 05:34:21 UTC
I'm seeing very similar corruption on MariaDB 10.4.21 after upgrading from 10.4.20 (same patch release as this bug is reported against, just backported to 10.4). MariaDB logs below:


2021-08-13 15:10:36 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-08-13 15:10:36 0 [Note] InnoDB: Uses event mutexes
2021-08-13 15:10:36 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2021-08-13 15:10:36 0 [Note] InnoDB: Number of pools: 1
2021-08-13 15:10:36 0 [Note] InnoDB: Using SSE2 crc32 instructions
2021-08-13 15:10:36 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2021-08-13 15:10:36 0 [Note] InnoDB: Completed initialization of buffer pool
2021-08-13 15:10:36 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=21524628584
2021-08-13 15:10:36 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
2021-08-13 15:10:36 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2021-08-13 15:10:36 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2021-08-13 15:10:36 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2021-08-13 15:10:36 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2021-08-13 15:10:36 0 [Note] InnoDB: Waiting for purge to start
2021-08-13 15:10:36 0 [Note] InnoDB: 10.4.21 started; log sequence number 21524628593; transaction id 33199647
2021-08-13 15:10:36 0 [Note] InnoDB: Loading buffer pool(s) from /var/db/mysql/ib_buffer_pool
2021-08-13 15:10:36 0 [Note] Plugin 'FEEDBACK' is disabled.
2021-08-13 15:10:36 0 [Note] Server socket created on IP: '0.0.0.0'.
2021-08-13 15:10:36 0 [Note] Reading of all Master_info entries succeeded
2021-08-13 15:10:36 0 [Note] Added new Master_info '' to hash table
2021-08-13 15:10:36 0 [Note] /usr/local/libexec/mysqld: ready for connections.
Version: '10.4.21-MariaDB'  socket: '/var/run/mysql/mysql.sock'  port: 3306  FreeBSD Ports
2021-08-13 15:10:36 0 [Note] InnoDB: Buffer pool(s) load completed at 210813 15:10:36
210813 15:10:49 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.

Server version: 10.4.21-MariaDB
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=1
max_threads=153
thread_count=6
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467689 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x823238e48
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7fffdc747f78 thread_stack 0x49000
0x128342c <my_print_stacktrace+0x3c> at /usr/local/libexec/mysqld
0xc1c4f5 <handle_fatal_signal+0x295> at /usr/local/libexec/mysqld
0x8017c7b70 <_pthread_sigmask+0x530> at /lib/libthr.so.3

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x82327c090): select count(*) from table_name_removed

Connection ID (thread ID): 10
Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on

The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Core pattern: %N.core
Comment 3 Bernard Spil freebsd_committer freebsd_triage 2021-08-13 11:57:02 UTC
What version/arch FreeBSD is this? make showconfig output?

Not sure if the same error, the second one for 10.4.21 is a segfault where the first doesn't seem to be a segfault.
Comment 4 Bernard Spil freebsd_committer freebsd_triage 2021-08-13 12:02:32 UTC
The 10.5.12 is 13.0-RELEASE-p3 amd64
InnoDB, Sphinx, Spider engines, ODBC/extra and LZ4 enabled

Are you using LZ4 compression in your DB? (CMake for LZ4 changed in 10.5.12)

The 10.4 sigsegv: is that built with GCC? and what arch? Options
Comment 5 Nathan Robertson 2021-08-13 22:58:13 UTC
12.2-RELEASE-p9 for me, running inside iocage jails on ZFS. Built out of poudriere. make.conf options are:

DEFAULT_VERSIONS += python=3.7 python3=3.7 mysql=10.4m pgsql=12 php=7.4 ssl=openssl
WANT_OPENLDAP_SASL=yes
OPTIONS_UNSET+= GSSAPI_BASE GSSAPI_NONE
OPTIONS_SET+= GSSAPI_HEIMDAL
security_sudo_UNSET+= GSSAPI_HEIMDAL

The first part of the poudriere build log is:

=>> Building databases/mariadb104-server
build started at Wed Aug 11 03:38:22 AEST 2021
port directory: /usr/ports/databases/mariadb104-server
package name: mariadb104-server-10.4.21
building for: FreeBSD blsyd-pkg.lesmills.net.au 12.2-RELEASE-p9 FreeBSD 12.2-RELEASE-p9 amd64
maintained by: brnrd@FreeBSD.org
Makefile ident: 
Poudriere version: 3.3.6
Host OSVERSION: 1202000
Jail OSVERSION: 1202000
Job Id: 02

---Begin Environment---
SHELL=/bin/csh
OSVERSION=1202000
UNAME_v=FreeBSD 12.2-RELEASE-p9
UNAME_r=12.2-RELEASE-p9
BLOCKSIZE=K
MAIL=/var/mail/root
STATUS=1
HOME=/root
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/root/bin
LOCALBASE=/usr/local
USER=root
LIBEXECPREFIX=/usr/local/libexec/poudriere
POUDRIERE_VERSION=3.3.6
MASTERMNT=/usr/local/poudriere/data/.m/pjail_12_0_amd64-default/ref
POUDRIERE_BUILD_TYPE=bulk
PACKAGE_BUILDING=yes
SAVED_TERM=screen
GID=0
UID=0
PWD=/usr/local/poudriere/data/.m/pjail_12_0_amd64-default/ref/.p/pool
P_PORTS_FEATURES=FLAVORS SELECTED_OPTIONS
MASTERNAME=pjail_12_0_amd64-default
SCRIPTPREFIX=/usr/local/share/poudriere
OLDPWD=/usr/local/poudriere/data/.m/pjail_12_0_amd64-default/ref/.p
SCRIPTPATH=/usr/local/share/poudriere/bulk.sh
POUDRIEREPATH=/usr/local/bin/poudriere
---End Environment---

---Begin Poudriere Port Flags/Env---
PORT_FLAGS=
PKGENV=
FLAVOR=
DEPENDS_ARGS=
MAKE_ARGS=
---End Poudriere Port Flags/Env---

---Begin OPTIONS List---
===> The following configuration options are available for mariadb104-server-10.4.21:
     CONNECT_EXTRA=on: Enable ODBC and XML in CONNECT engine
     DOCS=on: Build and/or install documentation
     WSREP=on: Build wsrep clustering
====> Optional page compression
     LZ4=off: LZ4 compression support
     LZO=off: LZO compression support
     SNAPPY=off: Snappy compression library support
     ZSTD=off: Zstandard compression support (RocksDB only)
====> Optional MariaDB storage engines
     INNOBASE=on: InnoDB default engine
     MROONGA=off: Mroonga Full Text Search engine
     OQGRAPH=off: Open Query Graph Computation engine
     ROCKSDB=off: RocksDB LSM engine (Alpha)
     SPHINX=on: SphinxSE engine
     SPIDER=on: Partitioning and XA-transactions engine
     TOKUDB=off: Fractal tree index tree data structure engine
====> Optional Mroonga features
     ZMQ=off: ZeroMQ support
     MSGPACK=off: MsgPack support
====> GSSAPI Security API support: you have to select exactly one of them
     GSSAPI_BASE=off: GSSAPI support via base system (needs Kerberos)
     GSSAPI_HEIMDAL=on: GSSAPI support via security/heimdal
     GSSAPI_MIT=off: GSSAPI support via security/krb5
     GSSAPI_NONE=off: Disable GSSAPI support
===> Use 'make config' to modify these settings
---End OPTIONS List---
Comment 6 Joachim 2021-08-31 00:35:27 UTC
I had the same problems. First, after updating to MariaDB 10.4.21, I had InnoDB corruptions. I then made a fresh install of MariaDB 10.5.12. When I started the server I received the following errors:

2021-08-30 23:05:27 0 [Note] InnoDB: Uses event mutexes
2021-08-30 23:05:27 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2021-08-30 23:05:27 0 [Note] InnoDB: Number of pools: 1
2021-08-30 23:05:27 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2021-08-30 23:05:27 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
2021-08-30 23:05:27 0 [Note] InnoDB: Completed initialization of buffer pool
2021-08-30 23:05:27 0 [Note] InnoDB: Doublewrite buffer not found: creating new
2021-08-30 23:05:27 0x801f66000  InnoDB: Assertion failure in file /var/ports/usr/ports/databases/mariadb105-server/work/mariadb-10.5.12/storage/innobase/buf/buf0dblwr.cc line 169
InnoDB: Failing assertion: id.page_no() == size
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
InnoDB: about forcing recovery.
210830 23:05:27 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.

Server version: 10.5.12-MariaDB
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=0
max_threads=153
thread_count=0
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467792 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x49000
0x130dc8c <my_print_stacktrace+0x3c> at /usr/local/libexec/mariadbd
0xc70fff <handle_fatal_signal+0x28f> at /usr/local/libexec/mariadbd
0x801943b70 <_pthread_sigmask+0x530> at /lib/libthr.so.3
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Core pattern: %N.core


I'm running FreeBSD 12.2.
Comment 7 Matthias Fechner freebsd_committer freebsd_triage 2021-09-03 07:21:21 UTC
I will disabled LZ4 compression and see if that fixes the problem.
Comment 8 Matthias Fechner freebsd_committer freebsd_triage 2021-09-05 06:49:22 UTC
I tried it now without any compress options.
Same problem, version 10.5.12 is crashed and also destroyes by this crash InnoDBs.

I had to restore the full database from a backup, so be very carefull here!

Version 10.5.11 seems to run fine.

I would rate this issue definitely as critical, as data is destroy.

It is crashing with signal 6 and 10, /var/log/messages has:
Aug 18 14:00:03 anny kernel: pid 7088 (mariadbd), jid 0, uid 88: exited on signal 6 (core dumped)
...
Aug 18 16:00:05 anny kernel: pid 38046 (mariadbd), jid 0, uid 88: exited on signal 10 (core dumped)
Comment 9 Matthias Fechner freebsd_committer freebsd_triage 2021-09-05 06:52:29 UTC
Found a similar error like Nathan in #2 reported:
210904  8:33:32 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.5.11-MariaDB
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=2
max_threads=153
thread_count=2
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467795 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x821c06b18
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7fffdfd55f38 thread_stack 0x49000
0x131a7dc <my_print_stacktrace+0x3c> at /usr/local/libexec/mariadbd
0xc7c64f <handle_fatal_signal+0x28f> at /usr/local/libexec/mariadbd
0x8018d8e00 <_pthread_sigmask+0x530> at /lib/libthr.so.3

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x821c33330): SELECT /*!40001 SQL_NO_CACHE */ `option_id`, `option_name`, `option_value`, `autoload` FROM `wp_options`

Connection ID (thread ID): 14
Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off

The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Core pattern: %N.core
Comment 10 Matthias Fechner freebsd_committer freebsd_triage 2021-09-07 07:31:54 UTC
Could it be, that the new version has new optimizations for the CPU which the old version does not had?

I ask, because my build server has another CPU compared to the machine I deploy this package.
Comment 11 Joachim 2021-09-10 06:48:07 UTC
I found this bug report in MariaDBs jira: https://jira.mariadb.org/projects/MDEV/issues/MDEV-26537?filter=allopenissues

Is this bug causing all the trouble?
Comment 12 Matthias Fechner freebsd_committer freebsd_triage 2021-09-10 07:11:27 UTC
(In reply to Joachim from comment #11)
Thank, that could really be related to the problem I see, but I do not get this with version 10.5.11, so if that was changed with version 12 it can really be related and maybe the cause of this problem.
Comment 13 Bernard Spil freebsd_committer freebsd_triage 2021-09-10 21:35:51 UTC
The patch linked from https://jira.mariadb.org/projects/MDEV/issues/MDEV-26537 does not cleanly apply on mariadb105-server.

If this is indeed the fix for the issue, I'd be happy to add it. Can someone backport this to the 10.3/4/5 ports and attach it here?

thanks!
Comment 14 commit-hook freebsd_committer freebsd_triage 2021-09-11 12:18:55 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=48fcfdf50f2638c9e9815454c1db6e2116f15e57

commit 48fcfdf50f2638c9e9815454c1db6e2116f15e57
Author:     Bernard Spil <brnrd@FreeBSD.org>
AuthorDate: 2021-09-11 12:09:02 +0000
Commit:     Bernard Spil <brnrd@FreeBSD.org>
CommitDate: 2021-09-11 12:18:23 +0000

    databases/mariadb105-server: Fix DB corruption

     * InnoDB corrupts files due to incorrect st_blksize calculation

    PR:             257728
    Reported by:    mfechner
    Obtained from:  https://jira.mariadb.org/projects/MDEV/issues/MDEV-26537
    MFH:            2021Q3

 databases/mariadb105-server/Makefile               |   2 +-
 .../mariadb105-server/files/patch-MDEV-26537 (new) | 109 +++++++++++++++++++++
 2 files changed, 110 insertions(+), 1 deletion(-)
Comment 15 Bernard Spil freebsd_committer freebsd_triage 2021-09-11 12:32:25 UTC
Hi Matthias, Joachim, Nathan,

Can you check the updated port and get let me know if this fixes the issue for you? Then I can push the fix to 2021Q3 as well.

Thanks in advance!
Comment 16 commit-hook freebsd_committer freebsd_triage 2021-09-12 12:04:41 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=c7054cfdf84443f275ef3979680e21c5ee61dee1

commit c7054cfdf84443f275ef3979680e21c5ee61dee1
Author:     Bernard Spil <brnrd@FreeBSD.org>
AuthorDate: 2021-09-12 12:02:39 +0000
Commit:     Bernard Spil <brnrd@FreeBSD.org>
CommitDate: 2021-09-12 12:04:19 +0000

    databases/mariadb103-server: Fix DB corruption

     * InnoDB corrupts files due to incorrect st_blksize calculation

     PR:            257728, 257958
     Reported by:   mfechner, iron udjin gmail com
     Obtained from: https://jira.mariadb.org/projects/MDEV/issues/MDEV-26537
     MFH:           2021Q3

 databases/mariadb103-server/Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Comment 17 commit-hook freebsd_committer freebsd_triage 2021-09-12 12:04:44 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=15c1622ad8fa4271d5f18831528af4d5b215e79e

commit 15c1622ad8fa4271d5f18831528af4d5b215e79e
Author:     Bernard Spil <brnrd@FreeBSD.org>
AuthorDate: 2021-09-12 12:00:07 +0000
Commit:     Bernard Spil <brnrd@FreeBSD.org>
CommitDate: 2021-09-12 12:04:19 +0000

    databases/mariadb104-server: Fix DB corruption

     * InnoDB corrupts files due to incorrect st_blksize calculation

    PR:             257728, 257958
    Reported by:    mfechner, iron udjin gmail com
    Obtained from:  https://jira.mariadb.org/projects/MDEV/issues/MDEV-26537
    MFH:            2021Q3

 .../mariadb103-server/files/patch-MDEV-26537 (new) | 126 +++++++++++++++++++++
 databases/mariadb104-server/Makefile               |   1 +
 .../mariadb104-server/files/patch-MDEV-26537 (new) | 126 +++++++++++++++++++++
 3 files changed, 253 insertions(+)
Comment 18 commit-hook freebsd_committer freebsd_triage 2021-09-12 12:09:47 UTC
A commit in branch 2021Q3 references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=a0f5b4b14018d4bce6a1bfdd8ca51c97b5617ca9

commit a0f5b4b14018d4bce6a1bfdd8ca51c97b5617ca9
Author:     Bernard Spil <brnrd@FreeBSD.org>
AuthorDate: 2021-09-11 12:09:02 +0000
Commit:     Bernard Spil <brnrd@FreeBSD.org>
CommitDate: 2021-09-12 12:06:00 +0000

    databases/mariadb105-server: Fix DB corruption

     * InnoDB corrupts files due to incorrect st_blksize calculation

    PR:             257728
    Reported by:    mfechner
    Obtained from:  https://jira.mariadb.org/projects/MDEV/issues/MDEV-26537
    MFH:            2021Q3

    (cherry picked from commit 48fcfdf50f2638c9e9815454c1db6e2116f15e57)

 databases/mariadb105-server/Makefile               |   2 +-
 .../mariadb105-server/files/patch-MDEV-26537 (new) | 109 +++++++++++++++++++++
 2 files changed, 110 insertions(+), 1 deletion(-)
Comment 19 commit-hook freebsd_committer freebsd_triage 2021-09-12 12:09:52 UTC
A commit in branch 2021Q3 references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=56dddf512791ce4c73a9ecb62f822732699afeaf

commit 56dddf512791ce4c73a9ecb62f822732699afeaf
Author:     Bernard Spil <brnrd@FreeBSD.org>
AuthorDate: 2021-09-12 12:00:07 +0000
Commit:     Bernard Spil <brnrd@FreeBSD.org>
CommitDate: 2021-09-12 12:06:33 +0000

    databases/mariadb104-server: Fix DB corruption

     * InnoDB corrupts files due to incorrect st_blksize calculation

    PR:             257728, 257958
    Reported by:    mfechner, iron udjin gmail com
    Obtained from:  https://jira.mariadb.org/projects/MDEV/issues/MDEV-26537
    MFH:            2021Q3

    (cherry picked from commit 15c1622ad8fa4271d5f18831528af4d5b215e79e)

 .../mariadb103-server/files/patch-MDEV-26537 (new) | 126 +++++++++++++++++++++
 databases/mariadb104-server/Makefile               |   1 +
 .../mariadb104-server/files/patch-MDEV-26537 (new) | 126 +++++++++++++++++++++
 3 files changed, 253 insertions(+)
Comment 20 commit-hook freebsd_committer freebsd_triage 2021-09-12 12:09:53 UTC
A commit in branch 2021Q3 references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=6b48974b1abf27d0c2f51c3bb0b730ac58b7bb4f

commit 6b48974b1abf27d0c2f51c3bb0b730ac58b7bb4f
Author:     Bernard Spil <brnrd@FreeBSD.org>
AuthorDate: 2021-09-12 12:02:39 +0000
Commit:     Bernard Spil <brnrd@FreeBSD.org>
CommitDate: 2021-09-12 12:07:53 +0000

    databases/mariadb103-server: Fix DB corruption

     * InnoDB corrupts files due to incorrect st_blksize calculation

     PR:            257728, 257958
     Reported by:   mfechner, iron udjin gmail com
     Obtained from: https://jira.mariadb.org/projects/MDEV/issues/MDEV-26537
     MFH:           2021Q3

    (cherry picked from commit c7054cfdf84443f275ef3979680e21c5ee61dee1)

 databases/mariadb103-server/Makefile | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
Comment 21 Bernard Spil freebsd_committer freebsd_triage 2021-09-12 12:12:08 UTC
Finally closing this, thanks for your help in tickets and testing!
Comment 22 Matthias Fechner freebsd_committer freebsd_triage 2021-09-15 05:43:16 UTC
Thanks a lot for including the fix.
I installed today the new version on one of the production servers.
Normally it takes some hours to one day to trigger the bug.

If you do not hear anything, the next 24 hours, then the bug should be fixed by this patch.
Comment 23 Matthias Fechner freebsd_committer freebsd_triage 2021-09-17 11:37:08 UTC
No crash so far, I would say, the added diff fixed the problem.
Thanks a lot for this!
Comment 24 Ivo Karabojkov 2021-10-01 10:08:20 UTC
I'm not sure if my issue is related to this but I find a lot of crashes with MariaDB 10.5.12_1:

Oct  1 12:34:58 hermes kernel: pid 22001 (mariadbd), jid 0, uid 88: exited on signal 10 (core dumped)
Oct  1 12:41:58 hermes kernel: pid 33620 (mariadbd), jid 0, uid 88: exited on signal 6 (core dumped)
Oct  1 12:42:15 hermes kernel: pid 47537 (mariadbd), jid 0, uid 88: exited on signal 6 (core dumped)
Oct  1 12:51:04 hermes kernel: pid 49305 (mariadbd), jid 0, uid 88: exited on signal 10 (core dumped)
Oct  1 12:53:59 hermes kernel: pid 64701 (mariadbd), jid 0, uid 88: exited on signal 10 (core dumped)
Oct  1 12:56:13 hermes kernel: pid 70048 (mariadbd), jid 0, uid 88: exited on signal 10 (core dumped)
Oct  1 12:59:39 hermes kernel: pid 76596 (mariadbd), jid 0, uid 88: exited on signal 6 (core dumped)

Here is part of mysqld.err:

2021-10-01 12:59:38 0x81f754e00  InnoDB: Assertion failure in file /wrkdirs/usr/ports/databases/mariadb105-server/work/mariadb-10.5.12/storage/innobase/rem/rem0rec.cc line 732
InnoDB: Failing assertion: !(len & 0x4000)
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
InnoDB: about forcing recovery.
211001 12:59:38 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.5.12-MariaDB
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=3
max_threads=153
thread_count=3
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467583 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x82ae06918
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7fffdb732f38 thread_stack 0x49000
0x12a520c <my_print_stacktrace+0x3c> at /usr/local/libexec/mariadbd
0xc3669f <handle_fatal_signal+0x28f> at /usr/local/libexec/mariadbd
0x80198fe00 <_pthread_sigmask+0x530> at /lib/libthr.so.3

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x82ae32f70): SELECT `data`
FROM `wla0i_session`
WHERE `session_id` = X'7663706973687335633675653137333131646336303672316a67'

Connection ID (thread ID): 64
Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off

The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Core pattern: %N.core
Comment 25 irukandji 2021-10-28 12:43:51 UTC
I can confirm this, suddenly my mariadb server started to core dump at start. 

2021-10-28 14:41:34 0 [Note] /usr/local/libexec/mysqld (mysqld 10.5.12-MariaDB-log) starting as process 86746 ...

There is no newer version in ports.

2021-10-28 14:39:14 0 [Warning] You need to use --log-bin to make --binlog-format work.
2021-10-28 14:39:14 0 [Warning] The parameter innodb_file_format is deprecated and has no effect. It may
be removed in future releases. See https://mariadb.com/kb/en/library/xtradbinnodb-file-format/
2021-10-28 14:39:14 0 [Warning] The parameter innodb_buffer_pool_instances is deprecated and has no effec
t.
2021-10-28 14:39:14 0 [Note] InnoDB: !!! innodb_force_recovery is set to 1 !!!
2021-10-28 14:39:14 0 [Note] InnoDB: Uses event mutexes
2021-10-28 14:39:14 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2021-10-28 14:39:14 0 [Note] InnoDB: Number of pools: 1
2021-10-28 14:39:14 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2021-10-28 14:39:14 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 13421
7728
2021-10-28 14:39:14 0 [Note] InnoDB: Completed initialization of buffer pool
2021-10-28 14:39:14 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=4776944071,4776944071
2021-10-28 14:39:14 0 [Note] InnoDB: 1 transaction(s) which must be rolled back or cleaned up in total 1
row operations to undo
2021-10-28 14:39:14 0 [Note] InnoDB: Trx id counter is 1818133
2021-10-28 14:39:14 0 [Note] InnoDB: Starting final batch to recover 196 pages from redo log.
2021-10-28 14:39:14 0 [Note] InnoDB: 128 rollback segments are active.
2021-10-28 14:39:14 0 [Note] InnoDB: Starting in background the rollback of recovered transactions
2021-10-28 14:39:14 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
211028 14:39:14 2021-10-28 14:39:14 0 [Note] InnoDB: Creating shared tablespace for temporary tables
[ERROR] mysqld got signal 10 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.5.12-MariaDB-log
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=0
max_threads=153
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467792 K  bytes of memory
2021-10-28 14:39:14 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file f
ull; Please wait ...
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x49000
2021-10-28 14:39:14 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
0x130cd6c <my_print_stacktrace+0x3c> at /usr/local/libexec/mariadbd
0xc6e6af <handle_fatal_signal+0x28f> at /usr/local/libexec/mariadbd
2021-10-28 14:39:14 0 [Note] InnoDB: 10.5.12 started; log sequence number 4803328560; transaction id 1818
134
2021-10-28 14:39:14 0 [Note] InnoDB: Loading buffer pool(s) from /data/mariadb/main/ib_buffer_pool
2021-10-28 14:39:14 0 [Note] Plugin 'FEEDBACK' is disabled.
0x801904e00 <_pthread_sigmask+0x530> at /lib/libthr.so.3
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Core pattern: %N.core