Bug 231697 - net/openmpi2: MPI_Send to self fails (or receive from self fails?)
Summary: net/openmpi2: MPI_Send to self fails (or receive from self fails?)
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: amd64 Any
: --- Affects Only Me
Assignee: Danilo Egea Gondolfo
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-09-24 21:41 UTC by russo
Modified: 2018-09-27 20:06 UTC (History)
0 users

See Also:
bugzilla: maintainer-feedback? (danilo)


Attachments
Simple test case that just does send/receives, fails with OpenMPI 2 or 3. (1.69 KB, text/plain)
2018-09-24 21:41 UTC, russo
no flags Details
Patch to fix OpenMPI2 by removing --enable-heterogeneous (395 bytes, patch)
2018-09-25 06:30 UTC, russo
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description russo 2018-09-24 21:41:27 UTC
Created attachment 197466 [details]
Simple test case that just does send/receives, fails with OpenMPI 2 or 3.

We have observed our code failing when built with OpenMPI 2.1 or 3.x on FreeBSD and also on one other Linux platform, and have tracked it down at least on BSD to a simple test case, in which it is observed that data sent via MPI_Send calls to the same processor that it's running on are not received by a corresponding MPI_Irecv, a use case that is supposed to be standard compliant *AND* which DOES work with OpenMPI 1.10 on the same machine.

My uname -a:
FreeBSD yyy.zzz 10.4-STABLE FreeBSD 10.4-STABLE #0 r327510: Tue Jan  2 21:52:13 MST 2018     xxx@yyy.zzz:/usr/obj/usr/src/sys/GENERIC  amd64

The attached test program will print BAD on each line where it is supposed to report that proc#N received something from proc#M when M==N, if compiled with OpenMPI 2.1.x or 3.x.  It will pass just fine with OpenMPI 1.10.

We have run this on a few OSen other than BSD including RHEL6, RHEL7, and OS X, and none have the same issue.  It does appear, however, that Ubuntu 18.04's OpenMPI 2.1.x has the same problem.

It is not at all clear where this problem lies, except that the symptom is that the receive requests do not in fact receive any data if the sender is the same processor.

To reproduce:
   /usr/local/mpi/openmpi2/bin/mpicc -o testBUG967 testBUG967.c
   /usr/local/mpi/openmpi2/bin/mpirun -np 2 ./testBUG967
On my machine, this gives the output:
0 posting receive 0 0x803fc78b0
0 posting receive 1 0x803fc78b4
0 sending to 0 value 1000
1 posting receive 0 0x803fc78b0
1 posting receive 1 0x803fc78b4
1 sending to 0 value 2000
1 sending to 1 value 2001
0 sending to 1 value 1001
0 wait source 0 count 0 
0 wait source 1 count 4 
0 procs_from 0 vals_from -1000 BAD BAD BAD 
0 procs_from 1 vals_from 2000   
1 wait source 1 count 0 
1 wait source 0 count 4 
1 procs_from 1 vals_from -1000 BAD BAD BAD 
1 procs_from 0 vals_from 1001   

When run instead with openmpi 1 it gives the output actually expected:
> /usr/local/mpi/openmpi/bin/mpicc -o testBUG967 testBUG967.c 
> /usr/local/mpi/openmpi/bin/mpirun -np 2 ./testBUG967 
1 posting receive 0 0x803e23ad8
1 posting receive 1 0x803e23adc
1 sending to 0 value 2000
0 posting receive 0 0x803e23ad8
0 posting receive 1 0x803e23adc
0 sending to 0 value 1000
1 sending to 1 value 2001
0 sending to 1 value 1001
1 wait source 1 count 4 
1 wait source 0 count 4 
1 procs_from 1 vals_from 2001   
0 wait source 0 count 4 
0 wait source 1 count 4 
0 procs_from 0 vals_from 1000   
1 procs_from 0 vals_from 1001   
0 procs_from 1 vals_from 2000   

I have tried it with varying --mca btl options (tcp,self; sm,self; vader,self) as well, and it always gets the failed receive issue with all of them unless I use OpenMPI 1.x.


Additional information:
> pkg info openmpi2
openmpi2-2.1.5
Name           : openmpi2
Version        : 2.1.5
Installed on   : Mon Sep 24 15:31:19 2018 MDT
Origin         : net/openmpi2
Architecture   : FreeBSD:10:amd64
Prefix         : /usr/local
Categories     : net parallel
Licenses       : BSD3CLAUSE
Maintainer     : danilo@FreeBSD.org
WWW            : http://www.open-mpi.org/
Comment        : High Performance Message Passing Library
Options        :
        DEBUG          : on
        IPV6           : on
        SLURM          : off
        TORQUE         : off
Shared Libs required:
        libhwloc.so.5
        libevent-2.1.so.6
        libevent_pthreads-2.1.so.6
        libquadmath.so.0
        libgcc_s.so.1
        libgfortran.so.4
        libmunge.so.2

> pkg info openmpi
openmpi-1.10.7_3
Name           : openmpi
Version        : 1.10.7_3
Installed on   : Wed Aug 22 23:44:37 2018 MDT
Origin         : net/openmpi
Architecture   : FreeBSD:10:amd64
Prefix         : /usr/local
Categories     : net parallel
Licenses       : BSD3CLAUSE
Maintainer     : danilo@FreeBSD.org
WWW            : http://www.open-mpi.org/
Comment        : High Performance Message Passing Library
Options        :
        IPV6           : on
        SLURM          : off
        TORQUE         : off
        VT             : off
Shared Libs required:
        libquadmath.so.0
        libevent_pthreads-2.1.so.6
        libevent-2.1.so.6
        libhwloc.so.5
        libgfortran.so.4
        libgcc_s.so.1

> pkg info hwloc
hwloc-1.11.11
Name           : hwloc
Version        : 1.11.11
Installed on   : Wed Sep 19 08:08:13 2018 MDT
Origin         : devel/hwloc
Architecture   : FreeBSD:10:amd64
Prefix         : /usr/local
Categories     : devel
Licenses       : BSD3CLAUSE
Maintainer     : phd_kimberlite@yahoo.co.jp
WWW            : http://www.open-mpi.org/projects/hwloc/
Comment        : Portable Hardware Locality software package
Options        :
        CAIRO          : off
        DOCS           : on
Shared Libs required:
        libxml2.so.2
        libpciaccess.so.0
Comment 1 russo 2018-09-25 06:29:33 UTC
Turns out that this bug in Ubuntu's OpenMPI was already reported with its cause:

https://bugs.launchpad.net/ubuntu/+source/openmpi/+bug/1731938

The issue is "--enable-heterogenous" --- adding this option to the configure for OpenMPI causes the send/receive failure when the receiver and sender are the same processor.

The fix would be trivial --- simply remove that option from the configure that FreeBSD's package uses.  I have confirmed that removing this option from net/openmpi2/Makefile renders the attached program correct.


I should note that this issue was reported to OpenMPI in 2013:

https://github.com/open-mpi/ompi/issues/171

and instead of removing the option they just say "works in some releases, not in others.  closing due to lack of interest" a few months ago after leaving it open for 5 years.

Respectfully request that "--enable-heterogenous" be removed from makefile options for both OpenMPI 2 and OpenMPI 3.
Comment 2 russo 2018-09-25 06:30:06 UTC
Created attachment 197477 [details]
Patch to fix OpenMPI2 by removing --enable-heterogeneous
Comment 3 russo 2018-09-25 14:29:40 UTC
One more comment:

The Github repo for OpenMPI now has this in its README:

--enable-heterogeneous
  Enable support for running on heterogeneous clusters (e.g., machines
  with different endian representations).  Heterogeneous support is
    disabled by default because it imposes a minor performance penalty.

  *** THIS FUNCTIONALITY IS CURRENTLY BROKEN - DO NOT USE ***


So this should really not be getting used in BSD's makefile, as it makes OpenMPI unreliable.
Comment 4 russo 2018-09-25 14:30:20 UTC
Ok, ONE more comment.  The README for OpenMPI at github is at:  https://github.com/open-mpi/ompi/blob/master/README
Comment 5 Danilo Egea Gondolfo freebsd_committer 2018-09-25 21:03:13 UTC
Hi, thank you for your research and analysis. I'll commit this fix.
Comment 6 commit-hook freebsd_committer 2018-09-25 22:22:44 UTC
A commit references this bug:

Author: danilo
Date: Tue Sep 25 22:22:38 UTC 2018
New revision: 480703
URL: https://svnweb.freebsd.org/changeset/ports/480703

Log:
  - Remove --enable-heterogeneous from CONFIGURE_ARGS, this option is broken and causing problems
    see https://github.com/open-mpi/ompi/blob/master/README#L1387

  while here

  - Bump PORTREVISION
  - Move USES to the proper section to make portlint happy

  PR:		231697
  Submitted by:	russo AT bogodyn.org

Changes:
  head/net/openmpi2/Makefile
Comment 7 commit-hook freebsd_committer 2018-09-25 22:25:48 UTC
A commit references this bug:

Author: danilo
Date: Tue Sep 25 22:25:36 UTC 2018
New revision: 480705
URL: https://svnweb.freebsd.org/changeset/ports/480705

Log:
   - Remove --enable-heterogeneous from CONFIGURE_ARGS, this option is broken and causing problems
     see https://github.com/open-mpi/ompi/blob/master/README#L1387

  while here

   - Bump PORTREVISION
   - Move USES to the proper section to make portlint happy

  PR:		231697
  Submitted by:	russo AT bogodyn.org

Changes:
  head/net/openmpi3/Makefile
Comment 8 russo 2018-09-26 20:46:51 UTC
Thank you for your prompt action on this bug.

I have confirmed that your change fixes OpenMPI 2 on the test code I posted.
Comment 9 Danilo Egea Gondolfo freebsd_committer 2018-09-27 20:06:14 UTC
Cool, thank you!