Bug 246182 - Kernel panic with sendfile() on ext2fs mounted filesystems
Summary: Kernel panic with sendfile() on ext2fs mounted filesystems
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 12.1-RELEASE
Hardware: Any Any
: --- Affects Only Me
Assignee: freebsd-bugs mailing list
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-05-04 17:43 UTC by Teran McKinney
Modified: 2020-05-25 14:44 UTC (History)
4 users (show)

See Also:


Attachments
core.txt.0 on i386 (77.21 KB, text/plain)
2020-05-07 18:43 UTC, Fedor Uporov
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Teran McKinney 2020-05-04 17:43:29 UTC
sendfile() with ext2fs can cause a kernel panic.

Tested on 12.1-RELEASE with x86_64 and ARMv7.

Steps:

1. Mount a filesystem with ext2fs.

2. open() a file under the mount point. Bigger files seem to work best, like
1GiB or so.

3. sendfile() that filedescriptor to the socket of your choice (127.0.0.1 on
some listening port that won't disconnect is fine, like nc -l 1234 >
/dev/null).

It seems to be kind of random for when the kernel panics, but it happens
inevitably. I've had it take anywhere from a second to maybe 10-20.  Data
speed seems to have an effect, but maybe it's just the total amount
transferred. I'm not sure.

A web server like nginx that gives access to files mounted with ext2fs can
trigger this if it's setup to use sendfile (I think most are). Or any user
with access to an ext2fs mounted partition can trigger it. Does not have
to be ran as root.

I don't know if this can be skillfully exploited to give something more
interesting than a kernel panic or not.

Sample code to help with testing:

#include <stdio.h>
#include <string.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/uio.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <netinet/tcp.h>
#include <fcntl.h>

char *self;

#define destinationPort 1234

int main(int argc, char **argv) {
    self=argv[0];
    if (argc != 2) {
        fprintf(stderr, "Usage: %s <file>\n", self);
        return(2);
    }
    int srcfp = open(argv[1], O_RDONLY);
    if (srcfp < 0) {
        perror("open");
        return(1);
    }

    int destinationSocket;
    if ((destinationSocket = socket(PF_INET, SOCK_STREAM, 0)) < 0) {
        perror("socket");
        return(1);
    }


    struct sockaddr_in sa;
    bzero(&sa, sizeof(sa));
    sa.sin_addr.s_addr = htonl(INADDR_LOOPBACK);
    sa.sin_family = AF_INET;
    sa.sin_port = htons(destinationPort);
    if (connect(destinationSocket, (struct sockaddr *)&sa, sizeof(sa)) < 0) {
        perror("connect");
        return(1);
    }

    if (sendfile(srcfp, destinationSocket, 0, 0, NULL, 0, 0) != 0) {
        perror("sendfile");
        return(1);
    }
    
    close(srcfp);
    close(destinationSocket);
    return(0);
}
Comment 1 Teran McKinney 2020-05-04 17:47:06 UTC
Is it possible that this is related to bug #238700?
Comment 2 Konstantin Belousov freebsd_committer 2020-05-04 17:53:37 UTC
Show the panic.

Also might be useful to test with HEAD kernel, where a lot of fixes for sendfile(2)
were done.
Comment 3 Teran McKinney 2020-05-04 19:56:52 UTC
Thanks for replying.

I have only had this happen with EXT2, EXT3, or EXT4. No other filesystem I've tested. FFS, ZFS, and are fine, likely FAT as well.

This is from a Beagleboard. I believe x86_64 gives similar Translation Fault panics.

root@generic:~ # ./sendfiler /mnt/FreeBSD-12.1-RELEASE-arm-armv7-BEAGLEBONE.img.xz
Fatal kernel mode data abort: 'Translation Fault (L2)' on read
trapframe: 0xc22c7970
FSR=00000007, FAR=00000042, spsr=80000013
r0 =c6554000, r1 =c1915694, r2 =00000022, r3 =bff195d0
r4 =00000000, r5 =00000000, r6 =00000022, r7 =bff19600
r8 =00000412, r9 =c0758c00, r10=00000020, r11=c22c7a20
r12=bff195d8, ssp=c22c7a04, slr=c08bf85c, pc =c060400c

panic: Fatal abort
cpuid = 0
time = 1572586114
Uptime: 1h32m49s
Automatic reboot in 15 seconds - press a key on the console to abort
Comment 4 Fedor Uporov freebsd_committer 2020-05-07 18:43:53 UTC
Created attachment 214241 [details]
core.txt.0 on i386
Comment 5 Fedor Uporov freebsd_committer 2020-05-07 18:45:12 UTC
Hi, Teran.

Thank you for your report and the reproducer snippet.

I should say that reproducer is really helpful.
The issue does not reproduced on 64bit architectures. I checked amd64 and aarch64, but could be reproduced on i386 (See core.txt in the attachment above).
Later I will try to investigate it.
Comment 6 Teran McKinney 2020-05-07 18:55:14 UTC
Hi Fedor,

Thank you so much for testing. I was having this on Intel x86_64 as well. You may just need a larger file to test with, or multiple runs. Nginx configured to use sendfile is another method I've used for testing. It's how I discovered the bug by accident.

If you need me to test anything please let me know.

Sincerely,

Teran
Comment 7 commit-hook freebsd_committer 2020-05-17 14:11:28 UTC
A commit references this bug:

Author: fsu
Date: Sun May 17 14:10:46 UTC 2020
New revision: 361135
URL: https://svnweb.freebsd.org/changeset/base/361135

Log:
  Restrict the max runp and runb return values in case of extents mapping.

  This restriction already present in case of indirect mapping, do the same
  in case of extents.

  PR:		246182
  Reported by:	Teran McKinney
  MFC after:	2 weeks

Changes:
  head/sys/fs/ext2fs/ext2_bmap.c
Comment 8 Fedor Uporov freebsd_committer 2020-05-17 14:13:17 UTC
Hi, Teran.

I made the commit, which should fix the panic reproduced by me using your snippet.
Could you please check it on your side if possible.
Comment 9 Teran McKinney 2020-05-18 00:45:57 UTC
Hi Fedor,

It works! Confirmed on x86_64 and ARMv7.

Really appreciate you taking the time to dive in and write up a fix. Thank you.

I don't know if FreeBSD 11 is affected as well or not, not sure if it would be worth backporting to 11 as well.

Let me know if you need anything else from me.
Comment 10 Fedor Uporov freebsd_committer 2020-05-25 14:44:34 UTC
Teran, thanks for verification.

You can email me directly if you will have any questions/issues with ext2fs.