OS X 32-bit wheels fail import in 1.15.0

[dhermes]

With the intel / fat python binaries provided by the python.org installers, trying to import in 32-bit mode fails with:

$ arch -32 /Library/Frameworks/Python.framework/Versions/3.6/bin/python -c "import numpy as np; print(np)"
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/numpy/core/__init__.py", line 16, in <module>
    from . import multiarray
ImportError: dlopen(/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/numpy/core/multiarray.cpython-36m-darwin.so, 2): Library not loaded: /usr/local/gfortran/lib/i386/libgfortran.3.dylib
  Referenced from: /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/numpy/.dylibs/libopenblasp-r0.3.0.dev.dylib
  Reason: image not found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/numpy/__init__.py", line 142, in <module>
    from . import add_newdocs
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/numpy/add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/numpy/lib/__init__.py", line 8, in <module>
    from .type_check import *
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/numpy/lib/type_check.py", line 11, in <module>
    import numpy.core.numeric as _nx
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/numpy/core/__init__.py", line 26, in <module>
    raise ImportError(msg)
ImportError:
Importing the multiarray numpy extension module failed.  Most
likely you are trying to import a failed build of numpy.
If you're working with a numpy git repo, try `git clean -xdf` (removes all
files not under version control).  Otherwise reinstall numpy.

Original error was: dlopen(/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/numpy/core/multiarray.cpython-36m-darwin.so, 2): Library not loaded: /usr/local/gfortran/lib/i386/libgfortran.3.dylib
  Referenced from: /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/numpy/.dylibs/libopenblasp-r0.3.0.dev.dylib
  Reason: image not found

Similar errors occur in 2.7 and 3.5.

[charris]

I don't think we provide 32 bit wheels for OS X. There was a recent attempt to build fat binaries, but it failed. @matthew-brett Comment?

[charris]

Does this work for earlier NumPy wheels?

[charris]

Does Python.org use Intel?

[rgommers]

$ arch -32

IIRC this has never worked properly, even when we were distributing fat binaries as DMG installers.

[dhermes]

Does this work for earlier NumPy wheels?

$ /Library/Frameworks/Python.framework/Versions/3.6/bin/python -m virtualenv venv
...
$ venv/bin/python -m pip install 'numpy < 1.15'
...
$ venv/bin/python -m pip freeze
numpy==1.14.5
$ arch -32 venv/bin/python -c "import numpy as np; print(np)"
<module 'numpy' from '.../venv/lib/python3.6/site-packages/numpy/__init__.py'>
$ arch -64 venv/bin/python -c "import numpy as np; print(np)"
<module 'numpy' from '.../venv/lib/python3.6/site-packages/numpy/__init__.py'>

and I can say empirically that I have been running unit tests for bezier in 32-bit OS X for many releases (1.13 and 1.14 series at least) and there have been no numpy issues.

Does Python.org use Intel?

This is just the nice name for "fat" binaries on OS X. For the latest official 3.6:

$ /Library/Frameworks/Python.framework/Versions/3.6/bin/python -V
Python 3.6.6

the binary supports both architectures:

$ file /Library/Frameworks/Python.framework/Versions/3.6/bin/python
/Library/Frameworks/Python.framework/Versions/3.6/bin/python: Mach-O universal binary with 2 architectures: [i386:Mach-O executable i386] [x86_64:Mach-O 64-bit executable x86_64]
/Library/Frameworks/Python.framework/Versions/3.6/bin/python (for architecture i386):   Mach-O executable i386
/Library/Frameworks/Python.framework/Versions/3.6/bin/python (for architecture x86_64): Mach-O 64-bit executable x86_64

[matthew-brett]

Hmm - this should work. In fact, I thought we were testing 32-bit numpy, see: https://github.com/MacPython/numpy-wheels/blob/master/config.sh#L38

[dhermes]

The /usr/local/gfortran/lib/i386/libgfortran.3.dylib in particular sticks out to me since a recent-ish update to the gcc provided by Homebrew upgraded this to libgfortran.4.dylib:

$ find /usr/local/Cellar | grep -i 'i386.*libgfortran\..*\.dylib$'
/usr/local/Cellar/gcc/8.1.0/lib/gcc/8/i386/libgfortran.5.dylib
/usr/local/Cellar/gcc/7.3.0_1/lib/gcc/7/i386/libgfortran.4.dylib
/usr/local/Cellar/gcc/7.2.0/lib/gcc/7/i386/libgfortran.4.dylib

Though it may just be a red herring. It seems the < 1.15 series did not ship their own dylibs

$ venv1/bin/pip freeze
numpy==1.14.5
$ find venv1 | grep -i dylib

but 1.15 does

$ venv2/bin/pip freeze
numpy==1.15.0
$ find venv2 | grep -i dylib
venv2/lib/python3.6/site-packages/numpy/.dylibs
venv2/lib/python3.6/site-packages/numpy/.dylibs/libgfortran.3.dylib
venv2/lib/python3.6/site-packages/numpy/.dylibs/libquadmath.0.dylib
venv2/lib/python3.6/site-packages/numpy/.dylibs/libopenblasp-r0.3.0.dev.dylib
venv2/lib/python3.6/site-packages/numpy/.dylibs/libgcc_s.1.dylib

This (i.e. shipping portable dylibs) mirrors the current approach that SciPy has.

[matthew-brett]

I suspect what's happening is the delocate utility is only finding, and including, the libraries needed for the 64-bit build.

[dhermes]

Good call @matthew-brett. The gcc and openblasp-r libraries support both:

$ lipo -info venv2/lib/python3.6/site-packages/numpy/.dylibs/libopenblasp-r0.3.0.dev.dylib
Architectures in the fat file: venv2/lib/python3.6/site-packages/numpy/.dylibs/libopenblasp-r0.3.0.dev.dylib are: i386 x86_64
$ lipo -info venv2/lib/python3.6/site-packages/numpy/.dylibs/libgcc_s.1.dylib
Architectures in the fat file: venv2/lib/python3.6/site-packages/numpy/.dylibs/libgcc_s.1.dylib are: x86_64 i386

but the gfortran and quadmath libraries do not:

$ lipo -info venv2/lib/python3.6/site-packages/numpy/.dylibs/libgfortran.3.dylib
Non-fat file: venv2/lib/python3.6/site-packages/numpy/.dylibs/libgfortran.3.dylib is architecture: x86_64
$ lipo -info venv2/lib/python3.6/site-packages/numpy/.dylibs/libquadmath.0.dylib
Non-fat file: venv2/lib/python3.6/site-packages/numpy/.dylibs/libquadmath.0.dylib is architecture: x86_64

The issue is that the gcc distribution has a separate i386 and x86_64 version of libgfortran on OS X (or at least the one shipped by Homebrew):

$ find /usr/local/Cellar | grep -i 'libgfortran\.5\.dylib$' | xargs lipo -info
Non-fat file: /usr/local/Cellar/gcc/8.1.0/lib/gcc/8/i386/libgfortran.5.dylib is architecture: i386
Non-fat file: /usr/local/Cellar/gcc/8.1.0/lib/gcc/8/libgfortran.5.dylib is architecture: x86_64

@matthew-brett may remember I actually raised this issue (matthew-brett/delocate#23) with delocate. When I build OS X wheels for bezier I cobble together a "frankenstein" libgfortran that is 32-bit and 64-bit.

[matthew-brett]

@dhermes - do you also need scipy to work on 32-bit?

I could try and address this in delocate, but 32-bit is dying fast, and I'd rather just patch the build process to put in the 32-bit library, after the initial delocate run.

[dhermes]

No I don't have any need for 32-bit, was just trying to support users that might. I don't have a dog in the fight and am happy to stop building 32-bit wheels if they aren't deemed useful. (The existence of a 32-bit manylinux image was in part guiding me.)

[charris]

@matthew-brett any progress on this front.?

[matthew-brett]

Sorry - no progress from me. The options are:

• give up on 32-bit support
• do post-process fixup to vendor the 32-bit libraries.

The second is a bit of work, maybe an hour or two. But - my strong suspicion is that that (maybe literally) no-one is using the 32-bit part of the build. Worth an email to the mailing list to check?

[charris]

@matthew-brett Worth an email, but I suspect that the 1.15.0 release has reached a much larger audience and so far the only complaint has come from @dhermes, and he doesn't actually use the 32 bit builds. Is it possible to build 32 bit specific wheels? That might be a fallback option if someone really needs them.

[matthew-brett]

I don't think there are any pure 32-bit Pythons out there, and the only practical way to get a 32-bit executable running is with:

arch -i386 python

where python points to system Python or Python.org Python. But, even then, those pythons still declare themselves to be dual architecture builds, when checking for wheels:

$ arch -i386 /usr/bin/python -c "import distutils.util; print(distutils.util.get_platform())"
macosx-10.12-intel

That means that they'll look for dual arch (intel) wheels on PyPI, which is what we currently claim to be providing (but, as of our shift to OpenBLAS, are not providing). So, there would be no point in -i386 wheels, because neither of system Python or Python.org Python would see them.

[rgommers]

+1 for dropping 32-bit, there's at most a few handfuls of users - not worth the effort or even the space in the binaries

[charris]

Going to close this, the consensus seems to be for dropping 32 bit support on the Mac. I'll add that as comment to the 1.15.1 release notes.

[dhermes]

SGTM, now I have some packaging code to delete 😀