Bug 280165 - mysqld SIGILL with mysql80-server-8.0.35_1, mysql81-server-8.1.0 and mysql84-server-8.4.0 with FreeBSD 14.1
Summary: mysqld SIGILL with mysql80-server-8.0.35_1, mysql81-server-8.1.0 and mysql84-...
Status: New
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: arm64 Any
: --- Affects Some People
Assignee: Mikael Urankar
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2024-07-06 22:44 UTC by John Lind
Modified: 2024-09-13 15:24 UTC (History)
4 users (show)

See Also:
linimon: maintainer-feedback?


Attachments
v0 (4.32 KB, patch)
2024-08-23 17:22 UTC, Mikael Urankar
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description John Lind 2024-07-06 22:44:39 UTC
For both mysql80-server and mysql81-server I get


024-07-06T22:32:23.099644Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2024-07-06T22:32:32Z UTC - mysqld got signal 4 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
Thread pointer: 0x7e44949d7000
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
[0x22eb778] +0x0
[0x22eb8dc] +0x0
[0x85650e48] +0x0
[0x85650594] +0x0
[0x841081a4] +0x0
[0x3092db8] +0x0
[0x2f95e2c] +0x0
[0x2f960c0] +0x0
[0x2f90ac8] +0x0
[0x2f92680] +0x0
[0x2f9e010] +0x0
[0x3045ee8] +0x0
[0x2d9bd20] +0x0
[0x2ba010c] +0x0
[0x2ba68f4] +0x0
[0x1fb8550] +0x0
[0x30f3b04] +0x0
[0x856463a0] +0x0
stack_bottom = 940f0e90 thread_stack 0x100000
0x2d508d8 <_Z19my_print_stacktracePKhm+0x13c> at /usr/local/libexec/mysqld
0x22eb778 <_Z18print_fatal_signali+0x25c> at /usr/local/libexec/mysqld
0x22eb8dc <handle_fatal_signal+0x40> at /usr/local/libexec/mysqld
0x85650e48 <pthread_sigmask+0x534> at /lib/libthr.so.3

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): 
Connection ID (thread ID): 1
Status: NOT_KILLED
...
/usr/local/etc/rc.d/mysql-server: WARNING: failed precmd routine for mysql

(well, obviously, the addresses very slightly between 8.1 and 8.0 - this is 8.0)

The ONLY change I've made in the my.cnf file is
innodb_buffer_pool_size         = 64M
(I mean, come on, how can I afford 1G on a Raspberry Pi 3Plus?  It only HAS 1GB of RAM)
which I have used successfully on FreeBSD 13.1 with mysql 5.7 which is not available in pkgs for FreeBSD 14.1


# uname -a
FreeBSD remo 14.1-RELEASE FreeBSD 14.1-RELEASE releng/14.1-n267679-10e31f0946d8 GENERIC arm64
Comment 1 John Lind 2024-08-04 21:43:05 UTC
I created a completely new image from FreeBSD-14.1-RELEASE-arm64-aarch64-RPI.img.xz and tried mysql81-server.  As before, I created a  my.cnf file that was the same as the example except I reduced the buffer size:
innodb_buffer_pool_size         = 128M
I got some errors pertaining to file ownership and permissions, which I worked through (chown mysql:mysql /var/db/mysql*).  Then this happened:

# service mysql-server onestart
2024-08-03T21:53:00.766292Z 0 [System] [MY-015017] [Server] MySQL Server Initialization - start.
2024-08-03T21:53:00.783242Z 0 [System] [MY-013169] [Server] /usr/local/libexec/mysqld (mysqld 8.1.0) initializing of server in progress as process 4349
mysqld: Error on delete of '/var/db/mysql/auto.cnf' (OS errno 2 - No such file or directory)
2024-08-03T21:53:00.888899Z 0 [Warning] [MY-010107] [Server] World-writable config file '/var/db/mysql/auto.cnf' has been removed.
2024-08-03T21:53:00.908502Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2024-08-03T21:53:08Z UTC - mysqld got signal 4 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
Thread pointer: 0x55cb57bdc000
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
[0x23052b0] +0x0
[0x2305414] +0x0
[0x84ae6e48] +0x0
[0x84ae6594] +0x0
[0x842f51a4] +0x0
[0x30b3580] +0x0
[0x2fb4720] +0x0
[0x2fb49b4] +0x0
[0x2faf394] +0x0
[0x2fb0f54] +0x0
[0x2fbc904] +0x0
[0x30666fc] +0x0
[0x2dba484] +0x0
[0x2bbc0d4] +0x0
[0x2bc24e0] +0x0
[0x1fbe6b0] +0x0
[0x31142bc] +0x0
[0x84adc3a0] +0x0
stack_bottom = 937e0e90 thread_stack 0x100000
0x2d6db20 <_Z19my_print_stacktracePKhm+0x13c> at /usr/local/libexec/mysqld
0x23052b0 <_Z18print_fatal_signali+0x25c> at /usr/local/libexec/mysqld
0x2305414 <handle_fatal_signal+0x40> at /usr/local/libexec/mysqld
0x84ae6e48 <pthread_sigmask+0x534> at /lib/libthr.so.3

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): 
Connection ID (thread ID): 1
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
/usr/local/etc/rc.d/mysql-server: WARNING: failed precmd routine for mysql
# 

At that point, I uninstalled mysql81-server-8.1.0 and installed mysql84-server and tried again, with the same my.cnf file.  This time, I still got SIGILL but with a very different looking backtrace.

# service mysql-server onestart
2024-08-03T22:03:13.218920Z 0 [System] [MY-015017] [Server] MySQL Server Initialization - start.
2024-08-03T22:03:13.230469Z 0 [System] [MY-013169] [Server] /usr/local/libexec/mysqld (mysqld 8.4.0) initializing of server in progress as process 4503
mysqld: Error on delete of '/var/db/mysql/auto.cnf' (OS errno 2 - No such file or directory)
2024-08-03T22:03:13.321074Z 0 [Warning] [MY-010107] [Server] World-writable config file '/var/db/mysql/auto.cnf' has been removed.
2024-08-03T22:03:13.342227Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2024-08-03T22:03:21Z UTC - mysqld got signal 4 ;
Signal SIGILL (Illegal trap) at address 0x905879ee
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
Thread pointer: 0x2c0b80581000
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 92b4de78 thread_stack 0x100000
 #0 0x24ea743 <unknown>
 #1 0x24ea30f <unknown>
 #2 0x85b68e5b handle_signal at /usr/src/lib/libthr/thread/thr_sig.c:298
 #3 0x85b68593 thr_sighandler at /usr/src/lib/libthr/thread/thr_sig.c:243

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): 
Connection ID (thread ID): 1
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
/usr/local/etc/rc.d/mysql-server: WARNING: failed precmd routine for mysql
# 

I tried going to the web page that the error recommends, but I can't even find the error log.  :(  I'm trying to follow their suggestions, but this is failing so early in the process that nothing really applies.
Comment 2 John Lind 2024-08-05 18:19:29 UTC
While we're at it, let me add that /var/db/mysql_secure and /var/db/mysql_tmpdir get created with the wrong ownship.  After installing, you have to do
# chown mysql:mysql /var/db/mysql_*
or you won't get very far at all.  I know I mentioned this, but it's easy to miss and probably relatively easy to fix in the package.
Comment 3 John Lind 2024-08-06 15:00:42 UTC
Well, Oracle just told me to go fish.  How can I help move this forward?  Isn't ANYONE trying to use MySQL with FreeBSD 14.1?
Comment 4 John Lind 2024-08-12 22:10:29 UTC
Upon encouragement from a post in the forums, I ran mysqld from gdb and got the following.  Note that I edited out a bunch of the gdb boilerplate for brevity.  I hacked the /use/local/etc/rc.c/mysql-server script to get the exact parameters it is being called with.  I set up options in .gdb according to the reference manual entry for using gdb with mysqld (https://dev.mysql.com/doc/refman/8.4/en/using-gdb-on-mysqld.html).  That gave me the following.


# gdb /usr/local/libexec/mysqld

GNU gdb (GDB) 14.1 [GDB v14.1 for FreeBSD]

...

This GDB was configured as "aarch64-portbld-freebsd14.0".

Type "show configuration" for configuration details.

...

Reading symbols from /usr/local/libexec/mysqld...

(No debugging symbols found in /usr/local/libexec/mysqld)

(gdb) run --defaults-extra-file=/usr/local/etc/mysql/my.cnf --initialize-insecure --basedir=/usr/local --datadir=/var/db/mysql --user=mysql
Starting program: /usr/local/libexec/mysqld --defaults-extra-file=/usr/local/etc/mysql/my.cnf --initialize-insecure --basedir=/usr/local --datadir=/var/db/mysql --user=mysql
[New LWP 100236 of process 8836]
2024-08-12T21:56:45.403943Z 0 [System] [MY-015017] [Server] MySQL Server Initialization - start.
2024-08-12T21:56:45.417480Z 0 [System] [MY-013169] [Server] /usr/local/libexec/mysqld (mysqld 8.4.0) initializing of server in progress as process 8836
mysqld: Error on delete of '/var/db/mysql/auto.cnf' (OS errno 2 - No such file or directory)
2024-08-12T21:56:45.511851Z 0 [Warning] [MY-010107] [Server] World-writable config file '/var/db/mysql/auto.cnf' has been removed.
[New LWP 100237 of process 8836]
2024-08-12T21:56:45.532560Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
[New LWP 100238 of process 8836]
[LWP 100238 of process 8836 exited]
[New LWP 100239 of process 8836]
[New LWP 100240 of process 8836]
[New LWP 100241 of process 8836]
[New LWP 100242 of process 8836]
[New LWP 100243 of process 8836]
[New LWP 100244 of process 8836]
[New LWP 100245 of process 8836]
[New LWP 100246 of process 8836]
[New LWP 100247 of process 8836]
[New LWP 100248 of process 8836]
[New LWP 100249 of process 8836]
[New LWP 100250 of process 8836]
[New LWP 100251 of process 8836]
[New LWP 100252 of process 8836]
[New LWP 100253 of process 8836]
[New LWP 100254 of process 8836]
[New LWP 100255 of process 8836]
[New LWP 100256 of process 8836]

Thread 3 received signal SIGILL, Illegal instruction.
Illegal trap.
[Switching to LWP 100237 of process 8836]
0x0000000003291848 in hardware::crc32_using_pclmul(unsigned char const*, unsigned long) ()
(gdb) 
#
Comment 5 John Lind 2024-08-14 19:19:17 UTC
Disassembly after the SIGILL gives, in part:
(gdb) disas
Dump of assembler code for function _ZN8hardware18crc32_using_pclmulEPKhm:
...
--Type <RET> for more, q to quit, c to continue without paging--
   0x0000000003291820 <+716>:   crc32cx w10, w10, x16
   0x0000000003291824 <+720>:   ldp     x15, x18, [x0, #320]
   0x0000000003291828 <+724>:   crc32cx w13, w13, x1
   0x000000000329182c <+728>:   ldp     x17, x16, [x0, #432]
   0x0000000003291830 <+732>:   fmov    d0, x10
   0x0000000003291834 <+736>:   crc32cx w13, w13, x15
   0x0000000003291838 <+740>:   crc32cx w10, w13, x18
   0x000000000329183c <+744>:   ldp     x13, x15, [x0, #448]
   0x0000000003291840 <+748>:   crc32cx w14, w14, x17
   0x0000000003291844 <+752>:   crc32cx w14, w14, x16
=> 0x0000000003291848 <+756>:   pmull   v0.1q, v1.1d, v0.1d
   0x000000000329184c <+760>:   fmov    d1, x10
   0x0000000003291850 <+764>:   crc32cx w10, w14, x13
   0x0000000003291854 <+768>:   ldp     x13, x14, [x0, #464]
   0x0000000003291858 <+772>:   crc32cx w10, w10, x15
   0x000000000329185c <+776>:   pmull   v1.1q, v2.1d, v1.1d
   0x0000000003291860 <+780>:   crc32cx w10, w10, x13
   0x0000000003291864 <+784>:   ldp     x15, x13, [x0, #480]
   0x0000000003291868 <+788>:   crc32cx w10, w10, x14
   0x000000000329186c <+792>:   eor     v0.16b, v1.16b, v0.16b
   0x0000000003291870 <+796>:   ldr     x14, [x0, #496]
   0x0000000003291874 <+800>:   add     x0, x0, #0x1f8
Comment 6 John Lind 2024-08-21 15:41:51 UTC
acheron (https://forums.freebsd.org/members/acheron.15063/) has identified the problem.  The return value expression for can_use_poly_mul() should be "return capabilities & HWCAP_PMULL" instead of "return capabilities & HWCAP_CRC32";  This appears to be a simple copying error since the incorrect return expression is the same as the prior function definition, where it is correct.  This code is at https://github.com/freebsd/freebsd-ports/blob/2024Q3/databases/mysql84-server/files/patch-storage_innobase_ut_crc32.cc#L20 .  Also, the package needs to create /var/db/mysql_tempdir and /var/db/mysql_security with the ownership of "mysql:mysql intead" of "root:wheel".  With these changes, mysqld starts and responds to queries.
Comment 7 John Lind 2024-08-21 19:31:49 UTC
I assume this change will need to be made for all three versions of the server currently offered for FreeBSD 14.1:
mysql80-server
mysql81-server
mysql84-server
While I haven't figured out yet why I can't compile them, I do have a system where I can test all three with regard to this one particular bug, if that's helpful.  If there's a full regression test suite, I'd certainly be happy to take a look at it.
Comment 8 Mikael Urankar freebsd_committer freebsd_triage 2024-08-23 17:22:42 UTC
Created attachment 253040 [details]
v0
Comment 9 John Lind 2024-08-23 20:29:41 UTC
Does that address the ownership issues for /var/db/mysql_tempdir and /var/db/mysql_security as well?
Comment 10 commit-hook freebsd_committer freebsd_triage 2024-08-26 08:14:50 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=90179a1242bc8aec6bd15491841049ee823ed320

commit 90179a1242bc8aec6bd15491841049ee823ed320
Author:     Mikael Urankar <mikael@FreeBSD.org>
AuthorDate: 2024-08-23 17:19:58 +0000
Commit:     Mikael Urankar <mikael@FreeBSD.org>
CommitDate: 2024-08-26 08:13:37 +0000

    databases/mysql84-server: Fix SIGILL on aarch64

    We want to check HWCAP_PMULL not HWCAP_CRC32.

    Approved by:    portmgr (trivial runtime fix)
    PR:             280165

 databases/mysql84-server/Makefile                                 | 2 +-
 databases/mysql84-server/files/patch-storage_innobase_ut_crc32.cc | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
Comment 11 commit-hook freebsd_committer freebsd_triage 2024-08-26 08:14:52 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=5a6ab005518a4351994494df094a0396bc1c0e2e

commit 5a6ab005518a4351994494df094a0396bc1c0e2e
Author:     Mikael Urankar <mikael@FreeBSD.org>
AuthorDate: 2024-08-23 17:22:00 +0000
Commit:     Mikael Urankar <mikael@FreeBSD.org>
CommitDate: 2024-08-26 08:13:37 +0000

    databases/mysql81-server: Fix SIGILL on aarch64

    We want to check HWCAP_PMULL not HWCAP_CRC32.

    Approved by:    portmgr (trivial runtime fix)
    PR:             280165

 databases/mysql81-server/Makefile                                 | 2 +-
 databases/mysql81-server/files/patch-storage_innobase_ut_crc32.cc | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
Comment 12 commit-hook freebsd_committer freebsd_triage 2024-08-26 08:14:53 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=16519bd204a0f2a9c46282591eaf29fd01b87721

commit 16519bd204a0f2a9c46282591eaf29fd01b87721
Author:     Mikael Urankar <mikael@FreeBSD.org>
AuthorDate: 2024-08-23 17:21:45 +0000
Commit:     Mikael Urankar <mikael@FreeBSD.org>
CommitDate: 2024-08-26 08:13:37 +0000

    databases/mysql80-server: Fix SIGILL on aarch64

    We want to check HWCAP_PMULL not HWCAP_CRC32.

    Approved by:    portmgr (trivial runtime fix)
    PR:             280165

 databases/mysql80-server/Makefile                                 | 2 +-
 databases/mysql80-server/files/patch-storage_innobase_ut_crc32.cc | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)