Bug 262378

CC author of suspected patch.

Comment 2 Mark Johnston 2022-03-06 13:19:50 UTC

(In reply to Damjan Jovanovic from comment #1)
Do you have ASLR enabled?  Can you show output from

$ sysctl kern.elf64
$ sysctl kern.elf32

?

Finished the git bisect and it confirms:

1811c1e957ee1250b08b3246fc0db37ddf64b736 is the first bad commit
commit 1811c1e957ee1250b08b3246fc0db37ddf64b736
Author: Mark Johnston <markj@FreeBSD.org>
Date    Mon Jan 17 11:42:56 2022 -0500

(In reply to Mark Johnston from comment #2)

It's still broken even in 46d35d415aa9664b0ddc98a76e453eec20af0016 (the vm_stacktop fix).

These are the sysctls:

kern.elf64.allow_wx: 1
kern.elf64.sigfastblock: 1
kern.elf64.aslr.stack: 1
kern.elf64.aslr.honor_sbrk: 0
kern.elf64.aslr.pie_enable: 1
kern.elf64.aslr.enable: 1
kern.elf64.pie_base: 16912384
kern.elf64.vdso: 1
kern.elf64.nxstack: 1
kern.elf64.fallback_brand: -1
kern.elf32.allow_wx: 1
kern.elf32.sigfastblock: 1
kern.elf32.aslr.stack: 1
kern.elf32.aslr.honor_sbrk: 0
kern.elf32.aslr.pie_enable: 0
kern.elf32.aslr.enable: 0
kern.elf32.pie_base: 16781312
kern.elf32.read_exec: 0
kern.elf32.vdso: 1
kern.elf32.nxstack: 1
kern.elf32.fallback_brand: -1

Still on 46d35d415aa9664b0ddc98a76e453eec20af0016, setting:
kern.elf64.aslr.stack=0
or
kern.elf64.aslr.enable=0
gets Wine working.

Comment 6 Stefan Eßer 2022-03-11 09:01:34 UTC

(In reply to Damjan Jovanovic from comment #5)

It is not necessary to disable ASLR for the whole system.

# elfctl +noaslr /usr/local/bin/wine64.bin

will disable ASLR just for this binary.
(Repeat for wine.bin ...)

Comment 7 Mark Johnston 2022-03-11 15:32:53 UTC

(In reply to Stefan Eßer from comment #6)
If ASLR is off as indicated by the sysctls, then kern.elfN.aslr.stack should not have an effect, so this is a kernel bug.  That is, I do not think elfctl is the right solution.

Sorry for the delayed follow-up, I am away from home.  I'll debug on or before Monday, if no one beats me to it.

Comment 8 Ed Maste 2022-03-11 15:55:04 UTC

(In reply to Mark Johnston from comment #7)

As far as I can tell the report is consistent with the stack enable control behaving as expected and wine being incompatible with our stack randomization; comment #4 shows

kern.elf64.aslr.stack: 1
kern.elf64.aslr.enable: 1

and comment #5 reports that setting either kern.elf64.aslr.stack or kern.elf64.aslr.enable to 0 results in wine working,

Comment 9 Mark Johnston 2022-03-11 16:04:51 UTC

Oh, bah.  I misread the sysctls in comment 4: ASLR is indeed enabled, I had been looking at the elf32 sysctls.  So yes, if wine is somehow incompatible with stack randomization (it would be best to understand how, exactly), then tagging it with elfctl is the right solution.  Sorry for the noise.

err:virtual:virtual_alloc_first_teb wine: failed to map the shared user data: c0000017

c0000017 == STATUS_NO_MEMORY

virtual_alloc_first_teb() in dlls/ntdll/unix/virtual.c gives that error when this fails:

---snip---
    /* reserve space for shared user data */
    status = NtAllocateVirtualMemory( NtCurrentProcess(),
        (void **)&user_shared_data, 0, &data_size,
        MEM_RESERVE | MEM_COMMIT, PAGE_READONLY );
---snip---

where, importantly:

struct _KUSER_SHARED_DATA *user_shared_data = (void *)0x7ffe0000;

That NtAllocateVirtualMemory() calls map_view() which calls map_fixed_area(), where mmap_is_in_reserved_area() returns 0 ("not in a reserved area, do a normal allocation"), leading to a call to anon_mmap_tryfixed(), which attempts to:

mmap((void *)0x7ffe0000, 4096, PROT_READ,
     MAP_PRIVATE|MAP_FIXED|MAP_ANON|MAP_EXCL, -1, 0);

This fails, setting off the error message and exit.

Why does it fail? Dumping /proc/curproc/map just before or after the mmap failure, shows this enormous mapping already occupying the required memory range (0x7ffe0000 - 0x7ffe1000):

0x62bd6000 0x82bb6000 0 0 0 --- 0 0 0x0 NCOW NNC none - NCH -1

It's 536,739,840 bytes (or 0x1ffe0000) long!

When I make a standalone binary that calls mmap() with those arguments, it succeeds, so that large mapping is not a standard feature of FreeBSD 14-CURRENT with ASLR. Where did it come from?

The large mapping exists even at the start of mmap_init() in that file, and at the start of its caller, virtual_init().

Starting at the beginning, and patching main() in loader/main.c to also dump /proc/curproc/map, shows that the large mapping exists even at the start of main()  :-(.

But why is that absent in my test binary?

Actually, my test binary also has that large mapping:

0x80063e000 0x82061e000 0 0 0 --- 0 0 0x0 NCOW NNC none - NCH -1

but it is situated much higher up in memory: starting at addresses around 0x80063e000 (beyond the 4th GiB), instead of around 0x62bd6000 for Wine, which frees up 0x7ffe0000 - 0x7ffe1000 which Wine needs.

Setting kern.elf64.aslr.stack=0 seems to make that large mapping go away, so it looks to be a FreeBSD 14-CURRENT ASLR thing after all.

Does anybody know why we allocate this 0x1ffe0000 byte long mapping when kern.elf64.aslr.stack=1, and what determines its placement in memory? Who does that, libc, rtld-elf, the kernel?

Comment 12 Ed Maste 2022-03-23 18:57:02 UTC

(In reply to Damjan Jovanovic from comment #11)
That large mapping is the stack space. It also exists with kern.elf64.aslr.stack=0, it's just fixed at the top of user memory (0x7fffdffff000).

A quick search turned up this commit for Mac: https://github.com/wine-mirror/wine/commit/ecd53057b5148cbe35fb67097d0063f796728e04
"libwine: On Mac, disable ASLR for Wine processes"
I think it's the same sort of issue.

Hello,

ASLR is now also in STABLE. Only wine-devel seems to work but can confirm that wine and wine-proton continue to work when disabling ASLR completely or by binary as mentioned by Damjan and Stefan, thank you!

Comment 14 Stefan Eßer 2022-07-24 21:30:09 UTC

(In reply to Alexander Vereeken from comment #13)

Thank you for reminding me that there are other Wine ports that have the same rum-time issue with ASLR:

I have applied the same elfctl -noalsr fixup to the binaries created by the emulators/wine and emulators/wine-proton ports.

Disabling ASLR on the Wine binaries is only a work-around, but the correct fix is much more complex and requires a new function to be implemented to provide suitable mapping addresses to be used by Wine when it allocates memory.

I have looked at the code that has been implemented to solve this issue on macOS, and it is not directly applicable to FreeBSD - therefore selectively exempting the Wine binaries from ASLR is the much quicker solution to a pressing problem ...

I'll close this PR, since the issue has been dealt with (admittedly in a  sub-optimal way, but the macOS patch mentioned in comment 12 does the same thing in a slightly different way).

If you think that disabling ASLR is not good enough, then feel free to open a feature request PR for the missing functionality that would allow ASLR to be enabled for Wine on FreeBSD.