Bug 262378 - emulators/wine-devel won't start on CURRENT due to ASLR changes
Summary: emulators/wine-devel won't start on CURRENT due to ASLR changes
Status: In Progress
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: CURRENT
Hardware: Any Any
: --- Affects Many People
Assignee: freebsd-bugs (Nobody)
URL:
Keywords: regression
Depends on:
Blocks:
 
Reported: 2022-03-06 12:44 UTC by Damjan Jovanovic
Modified: 2022-03-23 18:57 UTC (History)
7 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Damjan Jovanovic 2022-03-06 12:44:12 UTC
On recent CURRENT, Wine errors out and exits very quickly during startup:

$ wine64 cmd
err:virtual:virtual_alloc_first_teb wine: failed map the shared user data: c0000017

truss shows a lot of mmap() failures with errno 12.

"git bisect" between releng/13 and main has narrowed it down to this range of commits so far: dd694648ff0f..2075d00fabb

Reading over the commit messages for those commits, and toggling some of the sysctls mentioned, I found this one got it working again:

sysctl kern.elf64.aslr.stack=0

leading me to suspect the regression is from commit 1811c1e957ee1250b08b3246fc0db37ddf64b736 or thereabout.
Comment 1 Damjan Jovanovic 2022-03-06 12:46:22 UTC
CC author of suspected patch.
Comment 2 Mark Johnston freebsd_committer 2022-03-06 13:19:50 UTC
(In reply to Damjan Jovanovic from comment #1)
Do you have ASLR enabled?  Can you show output from

$ sysctl kern.elf64
$ sysctl kern.elf32

?
Comment 3 Damjan Jovanovic 2022-03-08 01:44:51 UTC
Finished the git bisect and it confirms:

1811c1e957ee1250b08b3246fc0db37ddf64b736 is the first bad commit
commit 1811c1e957ee1250b08b3246fc0db37ddf64b736
Author: Mark Johnston <markj@FreeBSD.org>
Date    Mon Jan 17 11:42:56 2022 -0500
Comment 4 Damjan Jovanovic 2022-03-08 05:15:20 UTC
(In reply to Mark Johnston from comment #2)

It's still broken even in 46d35d415aa9664b0ddc98a76e453eec20af0016 (the vm_stacktop fix).

These are the sysctls:

kern.elf64.allow_wx: 1
kern.elf64.sigfastblock: 1
kern.elf64.aslr.stack: 1
kern.elf64.aslr.honor_sbrk: 0
kern.elf64.aslr.pie_enable: 1
kern.elf64.aslr.enable: 1
kern.elf64.pie_base: 16912384
kern.elf64.vdso: 1
kern.elf64.nxstack: 1
kern.elf64.fallback_brand: -1
kern.elf32.allow_wx: 1
kern.elf32.sigfastblock: 1
kern.elf32.aslr.stack: 1
kern.elf32.aslr.honor_sbrk: 0
kern.elf32.aslr.pie_enable: 0
kern.elf32.aslr.enable: 0
kern.elf32.pie_base: 16781312
kern.elf32.read_exec: 0
kern.elf32.vdso: 1
kern.elf32.nxstack: 1
kern.elf32.fallback_brand: -1
Comment 5 Damjan Jovanovic 2022-03-08 05:26:55 UTC
Still on 46d35d415aa9664b0ddc98a76e453eec20af0016, setting:
kern.elf64.aslr.stack=0
or
kern.elf64.aslr.enable=0
gets Wine working.
Comment 6 Stefan Eßer freebsd_committer 2022-03-11 09:01:34 UTC
(In reply to Damjan Jovanovic from comment #5)

It is not necessary to disable ASLR for the whole system.

# elfctl +noaslr /usr/local/bin/wine64.bin

will disable ASLR just for this binary.
(Repeat for wine.bin ...)
Comment 7 Mark Johnston freebsd_committer 2022-03-11 15:32:53 UTC
(In reply to Stefan Eßer from comment #6)
If ASLR is off as indicated by the sysctls, then kern.elfN.aslr.stack should not have an effect, so this is a kernel bug.  That is, I do not think elfctl is the right solution.

Sorry for the delayed follow-up, I am away from home.  I'll debug on or before Monday, if no one beats me to it.
Comment 8 Ed Maste freebsd_committer 2022-03-11 15:55:04 UTC
(In reply to Mark Johnston from comment #7)

As far as I can tell the report is consistent with the stack enable control behaving as expected and wine being incompatible with our stack randomization; comment #4 shows

kern.elf64.aslr.stack: 1
kern.elf64.aslr.enable: 1

and comment #5 reports that setting either kern.elf64.aslr.stack or kern.elf64.aslr.enable to 0 results in wine working,
Comment 9 Mark Johnston freebsd_committer 2022-03-11 16:04:51 UTC
Oh, bah.  I misread the sysctls in comment 4: ASLR is indeed enabled, I had been looking at the elf32 sysctls.  So yes, if wine is somehow incompatible with stack randomization (it would be best to understand how, exactly), then tagging it with elfctl is the right solution.  Sorry for the noise.
Comment 10 Damjan Jovanovic 2022-03-23 17:26:52 UTC
err:virtual:virtual_alloc_first_teb wine: failed to map the shared user data: c0000017

c0000017 == STATUS_NO_MEMORY

virtual_alloc_first_teb() in dlls/ntdll/unix/virtual.c gives that error when this fails:

---snip---
    /* reserve space for shared user data */
    status = NtAllocateVirtualMemory( NtCurrentProcess(),
        (void **)&user_shared_data, 0, &data_size,
        MEM_RESERVE | MEM_COMMIT, PAGE_READONLY );
---snip---

where, importantly:

struct _KUSER_SHARED_DATA *user_shared_data = (void *)0x7ffe0000;

That NtAllocateVirtualMemory() calls map_view() which calls map_fixed_area(), where mmap_is_in_reserved_area() returns 0 ("not in a reserved area, do a normal allocation"), leading to a call to anon_mmap_tryfixed(), which attempts to:

mmap((void *)0x7ffe0000, 4096, PROT_READ,
     MAP_PRIVATE|MAP_FIXED|MAP_ANON|MAP_EXCL, -1, 0);

This fails, setting off the error message and exit.

Why does it fail? Dumping /proc/curproc/map just before or after the mmap failure, shows this enormous mapping already occupying the required memory range (0x7ffe0000 - 0x7ffe1000):

0x62bd6000 0x82bb6000 0 0 0 --- 0 0 0x0 NCOW NNC none - NCH -1

It's 536,739,840 bytes (or 0x1ffe0000) long!

When I make a standalone binary that calls mmap() with those arguments, it succeeds, so that large mapping is not a standard feature of FreeBSD 14-CURRENT with ASLR. Where did it come from?
Comment 11 Damjan Jovanovic 2022-03-23 18:05:08 UTC
The large mapping exists even at the start of mmap_init() in that file, and at the start of its caller, virtual_init().

Starting at the beginning, and patching main() in loader/main.c to also dump /proc/curproc/map, shows that the large mapping exists even at the start of main()  :-(.

But why is that absent in my test binary?

Actually, my test binary also has that large mapping:

0x80063e000 0x82061e000 0 0 0 --- 0 0 0x0 NCOW NNC none - NCH -1

but it is situated much higher up in memory: starting at addresses around 0x80063e000 (beyond the 4th GiB), instead of around 0x62bd6000 for Wine, which frees up 0x7ffe0000 - 0x7ffe1000 which Wine needs.

Setting kern.elf64.aslr.stack=0 seems to make that large mapping go away, so it looks to be a FreeBSD 14-CURRENT ASLR thing after all.

Does anybody know why we allocate this 0x1ffe0000 byte long mapping when kern.elf64.aslr.stack=1, and what determines its placement in memory? Who does that, libc, rtld-elf, the kernel?
Comment 12 Ed Maste freebsd_committer 2022-03-23 18:57:02 UTC
(In reply to Damjan Jovanovic from comment #11)
That large mapping is the stack space. It also exists with kern.elf64.aslr.stack=0, it's just fixed at the top of user memory (0x7fffdffff000).

A quick search turned up this commit for Mac: https://github.com/wine-mirror/wine/commit/ecd53057b5148cbe35fb67097d0063f796728e04
"libwine: On Mac, disable ASLR for Wine processes"
I think it's the same sort of issue.