Bug 257728
| Summary: | databases/mariadb105-server: 10.5.12 causes corrupted TRX_NO 488005401dea0a2 | ||
|---|---|---|---|
| Product: | Ports & Packages | Reporter: | Matthias Fechner <mfechner> |
| Component: | Individual Port(s) | Assignee: | Bernard Spil <brnrd> |
| Status: | Closed FIXED | ||
| Severity: | Affects Only Me | CC: | chris, irukandji, j.krieg, karabojkov, mfechner, nathan, peter |
| Priority: | --- | Flags: | bugzilla:
maintainer-feedback?
(brnrd) |
| Version: | Latest | ||
| Hardware: | Any | ||
| OS: | Any | ||
|
Description
Matthias Fechner
2021-08-10 07:17:45 UTC
I think the build logs are not important, but just to be complete. The build log for 10.5.11: https://pkg.fechner.net/data/130amd64-default/2021-08-10_08h57m30s/logs/mariadb105-server-10.5.11.log The build log for 10.5.12: https://pkg.fechner.net/data/130amd64-default/2021-08-09_10h05m12s/logs/mariadb105-server-10.5.12.log I'm seeing very similar corruption on MariaDB 10.4.21 after upgrading from 10.4.20 (same patch release as this bug is reported against, just backported to 10.4). MariaDB logs below: 2021-08-13 15:10:36 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2021-08-13 15:10:36 0 [Note] InnoDB: Uses event mutexes 2021-08-13 15:10:36 0 [Note] InnoDB: Compressed tables use zlib 1.2.11 2021-08-13 15:10:36 0 [Note] InnoDB: Number of pools: 1 2021-08-13 15:10:36 0 [Note] InnoDB: Using SSE2 crc32 instructions 2021-08-13 15:10:36 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M 2021-08-13 15:10:36 0 [Note] InnoDB: Completed initialization of buffer pool 2021-08-13 15:10:36 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=21524628584 2021-08-13 15:10:36 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2021-08-13 15:10:36 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2021-08-13 15:10:36 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2021-08-13 15:10:36 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2021-08-13 15:10:36 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2021-08-13 15:10:36 0 [Note] InnoDB: Waiting for purge to start 2021-08-13 15:10:36 0 [Note] InnoDB: 10.4.21 started; log sequence number 21524628593; transaction id 33199647 2021-08-13 15:10:36 0 [Note] InnoDB: Loading buffer pool(s) from /var/db/mysql/ib_buffer_pool 2021-08-13 15:10:36 0 [Note] Plugin 'FEEDBACK' is disabled. 2021-08-13 15:10:36 0 [Note] Server socket created on IP: '0.0.0.0'. 2021-08-13 15:10:36 0 [Note] Reading of all Master_info entries succeeded 2021-08-13 15:10:36 0 [Note] Added new Master_info '' to hash table 2021-08-13 15:10:36 0 [Note] /usr/local/libexec/mysqld: ready for connections. Version: '10.4.21-MariaDB' socket: '/var/run/mysql/mysql.sock' port: 3306 FreeBSD Ports 2021-08-13 15:10:36 0 [Note] InnoDB: Buffer pool(s) load completed at 210813 15:10:36 210813 15:10:49 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.21-MariaDB key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=1 max_threads=153 thread_count=6 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467689 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x823238e48 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fffdc747f78 thread_stack 0x49000 0x128342c <my_print_stacktrace+0x3c> at /usr/local/libexec/mysqld 0xc1c4f5 <handle_fatal_signal+0x295> at /usr/local/libexec/mysqld 0x8017c7b70 <_pthread_sigmask+0x530> at /lib/libthr.so.3 Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x82327c090): select count(*) from table_name_removed Connection ID (thread ID): 10 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains information that should help you find out what is causing the crash. Core pattern: %N.core What version/arch FreeBSD is this? make showconfig output? Not sure if the same error, the second one for 10.4.21 is a segfault where the first doesn't seem to be a segfault. The 10.5.12 is 13.0-RELEASE-p3 amd64 InnoDB, Sphinx, Spider engines, ODBC/extra and LZ4 enabled Are you using LZ4 compression in your DB? (CMake for LZ4 changed in 10.5.12) The 10.4 sigsegv: is that built with GCC? and what arch? Options 12.2-RELEASE-p9 for me, running inside iocage jails on ZFS. Built out of poudriere. make.conf options are: DEFAULT_VERSIONS += python=3.7 python3=3.7 mysql=10.4m pgsql=12 php=7.4 ssl=openssl WANT_OPENLDAP_SASL=yes OPTIONS_UNSET+= GSSAPI_BASE GSSAPI_NONE OPTIONS_SET+= GSSAPI_HEIMDAL security_sudo_UNSET+= GSSAPI_HEIMDAL The first part of the poudriere build log is: =>> Building databases/mariadb104-server build started at Wed Aug 11 03:38:22 AEST 2021 port directory: /usr/ports/databases/mariadb104-server package name: mariadb104-server-10.4.21 building for: FreeBSD blsyd-pkg.lesmills.net.au 12.2-RELEASE-p9 FreeBSD 12.2-RELEASE-p9 amd64 maintained by: brnrd@FreeBSD.org Makefile ident: Poudriere version: 3.3.6 Host OSVERSION: 1202000 Jail OSVERSION: 1202000 Job Id: 02 ---Begin Environment--- SHELL=/bin/csh OSVERSION=1202000 UNAME_v=FreeBSD 12.2-RELEASE-p9 UNAME_r=12.2-RELEASE-p9 BLOCKSIZE=K MAIL=/var/mail/root STATUS=1 HOME=/root PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/root/bin LOCALBASE=/usr/local USER=root LIBEXECPREFIX=/usr/local/libexec/poudriere POUDRIERE_VERSION=3.3.6 MASTERMNT=/usr/local/poudriere/data/.m/pjail_12_0_amd64-default/ref POUDRIERE_BUILD_TYPE=bulk PACKAGE_BUILDING=yes SAVED_TERM=screen GID=0 UID=0 PWD=/usr/local/poudriere/data/.m/pjail_12_0_amd64-default/ref/.p/pool P_PORTS_FEATURES=FLAVORS SELECTED_OPTIONS MASTERNAME=pjail_12_0_amd64-default SCRIPTPREFIX=/usr/local/share/poudriere OLDPWD=/usr/local/poudriere/data/.m/pjail_12_0_amd64-default/ref/.p SCRIPTPATH=/usr/local/share/poudriere/bulk.sh POUDRIEREPATH=/usr/local/bin/poudriere ---End Environment--- ---Begin Poudriere Port Flags/Env--- PORT_FLAGS= PKGENV= FLAVOR= DEPENDS_ARGS= MAKE_ARGS= ---End Poudriere Port Flags/Env--- ---Begin OPTIONS List--- ===> The following configuration options are available for mariadb104-server-10.4.21: CONNECT_EXTRA=on: Enable ODBC and XML in CONNECT engine DOCS=on: Build and/or install documentation WSREP=on: Build wsrep clustering ====> Optional page compression LZ4=off: LZ4 compression support LZO=off: LZO compression support SNAPPY=off: Snappy compression library support ZSTD=off: Zstandard compression support (RocksDB only) ====> Optional MariaDB storage engines INNOBASE=on: InnoDB default engine MROONGA=off: Mroonga Full Text Search engine OQGRAPH=off: Open Query Graph Computation engine ROCKSDB=off: RocksDB LSM engine (Alpha) SPHINX=on: SphinxSE engine SPIDER=on: Partitioning and XA-transactions engine TOKUDB=off: Fractal tree index tree data structure engine ====> Optional Mroonga features ZMQ=off: ZeroMQ support MSGPACK=off: MsgPack support ====> GSSAPI Security API support: you have to select exactly one of them GSSAPI_BASE=off: GSSAPI support via base system (needs Kerberos) GSSAPI_HEIMDAL=on: GSSAPI support via security/heimdal GSSAPI_MIT=off: GSSAPI support via security/krb5 GSSAPI_NONE=off: Disable GSSAPI support ===> Use 'make config' to modify these settings ---End OPTIONS List--- I had the same problems. First, after updating to MariaDB 10.4.21, I had InnoDB corruptions. I then made a fresh install of MariaDB 10.5.12. When I started the server I received the following errors: 2021-08-30 23:05:27 0 [Note] InnoDB: Uses event mutexes 2021-08-30 23:05:27 0 [Note] InnoDB: Compressed tables use zlib 1.2.11 2021-08-30 23:05:27 0 [Note] InnoDB: Number of pools: 1 2021-08-30 23:05:27 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions 2021-08-30 23:05:27 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728 2021-08-30 23:05:27 0 [Note] InnoDB: Completed initialization of buffer pool 2021-08-30 23:05:27 0 [Note] InnoDB: Doublewrite buffer not found: creating new 2021-08-30 23:05:27 0x801f66000 InnoDB: Assertion failure in file /var/ports/usr/ports/databases/mariadb105-server/work/mariadb-10.5.12/storage/innobase/buf/buf0dblwr.cc line 169 InnoDB: Failing assertion: id.page_no() == size InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to https://jira.mariadb.org/ InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/ InnoDB: about forcing recovery. 210830 23:05:27 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.5.12-MariaDB key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=0 max_threads=153 thread_count=0 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467792 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x0 thread_stack 0x49000 0x130dc8c <my_print_stacktrace+0x3c> at /usr/local/libexec/mariadbd 0xc70fff <handle_fatal_signal+0x28f> at /usr/local/libexec/mariadbd 0x801943b70 <_pthread_sigmask+0x530> at /lib/libthr.so.3 The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains information that should help you find out what is causing the crash. Core pattern: %N.core I'm running FreeBSD 12.2. I will disabled LZ4 compression and see if that fixes the problem. I tried it now without any compress options. Same problem, version 10.5.12 is crashed and also destroyes by this crash InnoDBs. I had to restore the full database from a backup, so be very carefull here! Version 10.5.11 seems to run fine. I would rate this issue definitely as critical, as data is destroy. It is crashing with signal 6 and 10, /var/log/messages has: Aug 18 14:00:03 anny kernel: pid 7088 (mariadbd), jid 0, uid 88: exited on signal 6 (core dumped) ... Aug 18 16:00:05 anny kernel: pid 38046 (mariadbd), jid 0, uid 88: exited on signal 10 (core dumped) Found a similar error like Nathan in #2 reported: 210904 8:33:32 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.5.11-MariaDB key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=2 max_threads=153 thread_count=2 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467795 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x821c06b18 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fffdfd55f38 thread_stack 0x49000 0x131a7dc <my_print_stacktrace+0x3c> at /usr/local/libexec/mariadbd 0xc7c64f <handle_fatal_signal+0x28f> at /usr/local/libexec/mariadbd 0x8018d8e00 <_pthread_sigmask+0x530> at /lib/libthr.so.3 Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x821c33330): SELECT /*!40001 SQL_NO_CACHE */ `option_id`, `option_name`, `option_value`, `autoload` FROM `wp_options` Connection ID (thread ID): 14 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains information that should help you find out what is causing the crash. Core pattern: %N.core Could it be, that the new version has new optimizations for the CPU which the old version does not had? I ask, because my build server has another CPU compared to the machine I deploy this package. I found this bug report in MariaDBs jira: https://jira.mariadb.org/projects/MDEV/issues/MDEV-26537?filter=allopenissues Is this bug causing all the trouble? (In reply to Joachim from comment #11) Thank, that could really be related to the problem I see, but I do not get this with version 10.5.11, so if that was changed with version 12 it can really be related and maybe the cause of this problem. The patch linked from https://jira.mariadb.org/projects/MDEV/issues/MDEV-26537 does not cleanly apply on mariadb105-server. If this is indeed the fix for the issue, I'd be happy to add it. Can someone backport this to the 10.3/4/5 ports and attach it here? thanks! A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=48fcfdf50f2638c9e9815454c1db6e2116f15e57 commit 48fcfdf50f2638c9e9815454c1db6e2116f15e57 Author: Bernard Spil <brnrd@FreeBSD.org> AuthorDate: 2021-09-11 12:09:02 +0000 Commit: Bernard Spil <brnrd@FreeBSD.org> CommitDate: 2021-09-11 12:18:23 +0000 databases/mariadb105-server: Fix DB corruption * InnoDB corrupts files due to incorrect st_blksize calculation PR: 257728 Reported by: mfechner Obtained from: https://jira.mariadb.org/projects/MDEV/issues/MDEV-26537 MFH: 2021Q3 databases/mariadb105-server/Makefile | 2 +- .../mariadb105-server/files/patch-MDEV-26537 (new) | 109 +++++++++++++++++++++ 2 files changed, 110 insertions(+), 1 deletion(-) Hi Matthias, Joachim, Nathan, Can you check the updated port and get let me know if this fixes the issue for you? Then I can push the fix to 2021Q3 as well. Thanks in advance! A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=c7054cfdf84443f275ef3979680e21c5ee61dee1 commit c7054cfdf84443f275ef3979680e21c5ee61dee1 Author: Bernard Spil <brnrd@FreeBSD.org> AuthorDate: 2021-09-12 12:02:39 +0000 Commit: Bernard Spil <brnrd@FreeBSD.org> CommitDate: 2021-09-12 12:04:19 +0000 databases/mariadb103-server: Fix DB corruption * InnoDB corrupts files due to incorrect st_blksize calculation PR: 257728, 257958 Reported by: mfechner, iron udjin gmail com Obtained from: https://jira.mariadb.org/projects/MDEV/issues/MDEV-26537 MFH: 2021Q3 databases/mariadb103-server/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=15c1622ad8fa4271d5f18831528af4d5b215e79e commit 15c1622ad8fa4271d5f18831528af4d5b215e79e Author: Bernard Spil <brnrd@FreeBSD.org> AuthorDate: 2021-09-12 12:00:07 +0000 Commit: Bernard Spil <brnrd@FreeBSD.org> CommitDate: 2021-09-12 12:04:19 +0000 databases/mariadb104-server: Fix DB corruption * InnoDB corrupts files due to incorrect st_blksize calculation PR: 257728, 257958 Reported by: mfechner, iron udjin gmail com Obtained from: https://jira.mariadb.org/projects/MDEV/issues/MDEV-26537 MFH: 2021Q3 .../mariadb103-server/files/patch-MDEV-26537 (new) | 126 +++++++++++++++++++++ databases/mariadb104-server/Makefile | 1 + .../mariadb104-server/files/patch-MDEV-26537 (new) | 126 +++++++++++++++++++++ 3 files changed, 253 insertions(+) A commit in branch 2021Q3 references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=a0f5b4b14018d4bce6a1bfdd8ca51c97b5617ca9 commit a0f5b4b14018d4bce6a1bfdd8ca51c97b5617ca9 Author: Bernard Spil <brnrd@FreeBSD.org> AuthorDate: 2021-09-11 12:09:02 +0000 Commit: Bernard Spil <brnrd@FreeBSD.org> CommitDate: 2021-09-12 12:06:00 +0000 databases/mariadb105-server: Fix DB corruption * InnoDB corrupts files due to incorrect st_blksize calculation PR: 257728 Reported by: mfechner Obtained from: https://jira.mariadb.org/projects/MDEV/issues/MDEV-26537 MFH: 2021Q3 (cherry picked from commit 48fcfdf50f2638c9e9815454c1db6e2116f15e57) databases/mariadb105-server/Makefile | 2 +- .../mariadb105-server/files/patch-MDEV-26537 (new) | 109 +++++++++++++++++++++ 2 files changed, 110 insertions(+), 1 deletion(-) A commit in branch 2021Q3 references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=56dddf512791ce4c73a9ecb62f822732699afeaf commit 56dddf512791ce4c73a9ecb62f822732699afeaf Author: Bernard Spil <brnrd@FreeBSD.org> AuthorDate: 2021-09-12 12:00:07 +0000 Commit: Bernard Spil <brnrd@FreeBSD.org> CommitDate: 2021-09-12 12:06:33 +0000 databases/mariadb104-server: Fix DB corruption * InnoDB corrupts files due to incorrect st_blksize calculation PR: 257728, 257958 Reported by: mfechner, iron udjin gmail com Obtained from: https://jira.mariadb.org/projects/MDEV/issues/MDEV-26537 MFH: 2021Q3 (cherry picked from commit 15c1622ad8fa4271d5f18831528af4d5b215e79e) .../mariadb103-server/files/patch-MDEV-26537 (new) | 126 +++++++++++++++++++++ databases/mariadb104-server/Makefile | 1 + .../mariadb104-server/files/patch-MDEV-26537 (new) | 126 +++++++++++++++++++++ 3 files changed, 253 insertions(+) A commit in branch 2021Q3 references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=6b48974b1abf27d0c2f51c3bb0b730ac58b7bb4f commit 6b48974b1abf27d0c2f51c3bb0b730ac58b7bb4f Author: Bernard Spil <brnrd@FreeBSD.org> AuthorDate: 2021-09-12 12:02:39 +0000 Commit: Bernard Spil <brnrd@FreeBSD.org> CommitDate: 2021-09-12 12:07:53 +0000 databases/mariadb103-server: Fix DB corruption * InnoDB corrupts files due to incorrect st_blksize calculation PR: 257728, 257958 Reported by: mfechner, iron udjin gmail com Obtained from: https://jira.mariadb.org/projects/MDEV/issues/MDEV-26537 MFH: 2021Q3 (cherry picked from commit c7054cfdf84443f275ef3979680e21c5ee61dee1) databases/mariadb103-server/Makefile | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) Finally closing this, thanks for your help in tickets and testing! Thanks a lot for including the fix. I installed today the new version on one of the production servers. Normally it takes some hours to one day to trigger the bug. If you do not hear anything, the next 24 hours, then the bug should be fixed by this patch. No crash so far, I would say, the added diff fixed the problem. Thanks a lot for this! I'm not sure if my issue is related to this but I find a lot of crashes with MariaDB 10.5.12_1: Oct 1 12:34:58 hermes kernel: pid 22001 (mariadbd), jid 0, uid 88: exited on signal 10 (core dumped) Oct 1 12:41:58 hermes kernel: pid 33620 (mariadbd), jid 0, uid 88: exited on signal 6 (core dumped) Oct 1 12:42:15 hermes kernel: pid 47537 (mariadbd), jid 0, uid 88: exited on signal 6 (core dumped) Oct 1 12:51:04 hermes kernel: pid 49305 (mariadbd), jid 0, uid 88: exited on signal 10 (core dumped) Oct 1 12:53:59 hermes kernel: pid 64701 (mariadbd), jid 0, uid 88: exited on signal 10 (core dumped) Oct 1 12:56:13 hermes kernel: pid 70048 (mariadbd), jid 0, uid 88: exited on signal 10 (core dumped) Oct 1 12:59:39 hermes kernel: pid 76596 (mariadbd), jid 0, uid 88: exited on signal 6 (core dumped) Here is part of mysqld.err: 2021-10-01 12:59:38 0x81f754e00 InnoDB: Assertion failure in file /wrkdirs/usr/ports/databases/mariadb105-server/work/mariadb-10.5.12/storage/innobase/rem/rem0rec.cc line 732 InnoDB: Failing assertion: !(len & 0x4000) InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to https://jira.mariadb.org/ InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/ InnoDB: about forcing recovery. 211001 12:59:38 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.5.12-MariaDB key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=3 max_threads=153 thread_count=3 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467583 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x82ae06918 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fffdb732f38 thread_stack 0x49000 0x12a520c <my_print_stacktrace+0x3c> at /usr/local/libexec/mariadbd 0xc3669f <handle_fatal_signal+0x28f> at /usr/local/libexec/mariadbd 0x80198fe00 <_pthread_sigmask+0x530> at /lib/libthr.so.3 Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x82ae32f70): SELECT `data` FROM `wla0i_session` WHERE `session_id` = X'7663706973687335633675653137333131646336303672316a67' Connection ID (thread ID): 64 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains information that should help you find out what is causing the crash. Core pattern: %N.core I can confirm this, suddenly my mariadb server started to core dump at start. 2021-10-28 14:41:34 0 [Note] /usr/local/libexec/mysqld (mysqld 10.5.12-MariaDB-log) starting as process 86746 ... There is no newer version in ports. 2021-10-28 14:39:14 0 [Warning] You need to use --log-bin to make --binlog-format work. 2021-10-28 14:39:14 0 [Warning] The parameter innodb_file_format is deprecated and has no effect. It may be removed in future releases. See https://mariadb.com/kb/en/library/xtradbinnodb-file-format/ 2021-10-28 14:39:14 0 [Warning] The parameter innodb_buffer_pool_instances is deprecated and has no effec t. 2021-10-28 14:39:14 0 [Note] InnoDB: !!! innodb_force_recovery is set to 1 !!! 2021-10-28 14:39:14 0 [Note] InnoDB: Uses event mutexes 2021-10-28 14:39:14 0 [Note] InnoDB: Compressed tables use zlib 1.2.11 2021-10-28 14:39:14 0 [Note] InnoDB: Number of pools: 1 2021-10-28 14:39:14 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions 2021-10-28 14:39:14 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 13421 7728 2021-10-28 14:39:14 0 [Note] InnoDB: Completed initialization of buffer pool 2021-10-28 14:39:14 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=4776944071,4776944071 2021-10-28 14:39:14 0 [Note] InnoDB: 1 transaction(s) which must be rolled back or cleaned up in total 1 row operations to undo 2021-10-28 14:39:14 0 [Note] InnoDB: Trx id counter is 1818133 2021-10-28 14:39:14 0 [Note] InnoDB: Starting final batch to recover 196 pages from redo log. 2021-10-28 14:39:14 0 [Note] InnoDB: 128 rollback segments are active. 2021-10-28 14:39:14 0 [Note] InnoDB: Starting in background the rollback of recovered transactions 2021-10-28 14:39:14 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 211028 14:39:14 2021-10-28 14:39:14 0 [Note] InnoDB: Creating shared tablespace for temporary tables [ERROR] mysqld got signal 10 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.5.12-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=0 max_threads=153 thread_count=0 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467792 K bytes of memory 2021-10-28 14:39:14 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file f ull; Please wait ... Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x0 thread_stack 0x49000 2021-10-28 14:39:14 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 0x130cd6c <my_print_stacktrace+0x3c> at /usr/local/libexec/mariadbd 0xc6e6af <handle_fatal_signal+0x28f> at /usr/local/libexec/mariadbd 2021-10-28 14:39:14 0 [Note] InnoDB: 10.5.12 started; log sequence number 4803328560; transaction id 1818 134 2021-10-28 14:39:14 0 [Note] InnoDB: Loading buffer pool(s) from /data/mariadb/main/ib_buffer_pool 2021-10-28 14:39:14 0 [Note] Plugin 'FEEDBACK' is disabled. 0x801904e00 <_pthread_sigmask+0x530> at /lib/libthr.so.3 The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains information that should help you find out what is causing the crash. Core pattern: %N.core |