pandas 0.23.2 fails to build on top of Python 3.7.0 (while 0.23.1 works fine)

[boegel]

After seeing the release announcement for pandas 0.23.2 where is was mentioned that this is the first release supporting Python 3.7.0, I looked into switching from pandas 0.23.1 which we were planning to use to 0.23.2 (cfr. easybuilders/easybuild-easyconfigs#6480).

However, I'm seeing compilation problems with pandas 0.23.2 on top of (a self-built) Python 3.7.0 (which are not there when building/installing pandas 0.23.1):

$ python -V
Python 3.7.0
$ cd pandas-0.23.2
$ python setup.py build
...
running build_ext
skipping 'pandas/_libs/algos.c' Cython extension (up-to-date)
building 'pandas._libs.algos' extension
creating build/temp.linux-x86_64-3.7
creating build/temp.linux-x86_64-3.7/pandas
creating build/temp.linux-x86_64-3.7/pandas/_libs
gcc -DNDEBUG -g -fwrapv -O3 -Wall -O2 -ftree-vectorize -march=native -fno-math-errno -fPIC -I/software/OpenBLAS/0.3.1-GCC-7.3.0-2.30/include -I/software/ScaLAPACK/2.0.2-gompi-2018b-OpenBLAS-0.3.1/include -I/software/FFTW/3.3.8-gompi-2018b/include -fPIC -Ipandas/_libs/src/klib -Ipandas/_libs/src -I/software/Python/3.7.0-foss-2018b/lib/python3.7/site-packages/numpy-1.14.5-py3.7-linux-x86_64.egg/numpy/core/include -I/software/Python/3.7.0-foss-2018b/include/python3.7m -c pandas/_libs/algos.c -o build/temp.linux-x86_64-3.7/pandas/_libs/algos.o -Wno-unused-function
In file included from /software/Python/3.7.0-foss-2018b/lib/python3.7/site-packages/numpy-1.14.5-py3.7-linux-x86_64.egg/numpy/core/include/numpy/ndarraytypes.h:1816:0,
                 from /software/Python/3.7.0-foss-2018b/lib/python3.7/site-packages/numpy-1.14.5-py3.7-linux-x86_64.egg/numpy/core/include/numpy/ndarrayobject.h:18,
                 from /software/Python/3.7.0-foss-2018b/lib/python3.7/site-packages/numpy-1.14.5-py3.7-linux-x86_64.egg/numpy/core/include/numpy/arrayobject.h:4,
                 from pandas/_libs/algos.c:435:
/software/Python/3.7.0-foss-2018b/lib/python3.7/site-packages/numpy-1.14.5-py3.7-linux-x86_64.egg/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
 #warning "Using deprecated NumPy API, disable it by " \
  ^~~~~~~
pandas/_libs/algos.c: In function ‘__Pyx__ExceptionSave’:
pandas/_libs/algos.c:138911:21: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_type’; did you mean ‘curexc_type’?
     *type = tstate->exc_type;
                     ^~~~~~~~
                     curexc_type
pandas/_libs/algos.c:138912:22: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_value’; did you mean ‘curexc_value’?
     *value = tstate->exc_value;
                      ^~~~~~~~~
                      curexc_value
pandas/_libs/algos.c:138913:19: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_traceback’; did you mean ‘curexc_traceback’?
     *tb = tstate->exc_traceback;
                   ^~~~~~~~~~~~~
                   curexc_traceback
pandas/_libs/algos.c: In function ‘__Pyx__ExceptionReset’:
pandas/_libs/algos.c:138920:24: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_type’; did you mean ‘curexc_type’?
     tmp_type = tstate->exc_type;
                        ^~~~~~~~
                        curexc_type
pandas/_libs/algos.c:138921:25: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_value’; did you mean ‘curexc_value’?
     tmp_value = tstate->exc_value;
                         ^~~~~~~~~
                         curexc_value
pandas/_libs/algos.c:138922:22: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_traceback’; did you mean ‘curexc_traceback’?
     tmp_tb = tstate->exc_traceback;
                      ^~~~~~~~~~~~~
                      curexc_traceback
pandas/_libs/algos.c:138923:13: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_type’; did you mean ‘curexc_type’?
     tstate->exc_type = type;
             ^~~~~~~~
             curexc_type
pandas/_libs/algos.c:138924:13: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_value’; did you mean ‘curexc_value’?
     tstate->exc_value = value;
             ^~~~~~~~~
             curexc_value
pandas/_libs/algos.c:138925:13: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_traceback’; did you mean ‘curexc_traceback’?
     tstate->exc_traceback = tb;
             ^~~~~~~~~~~~~
             curexc_traceback
pandas/_libs/algos.c: In function ‘__Pyx__GetException’:
pandas/_libs/algos.c:138980:24: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_type’; did you mean ‘curexc_type’?
     tmp_type = tstate->exc_type;
                        ^~~~~~~~
                        curexc_type
pandas/_libs/algos.c:138981:25: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_value’; did you mean ‘curexc_value’?
     tmp_value = tstate->exc_value;
                         ^~~~~~~~~
                         curexc_value
pandas/_libs/algos.c:138982:22: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_traceback’; did you mean ‘curexc_traceback’?
     tmp_tb = tstate->exc_traceback;
                      ^~~~~~~~~~~~~
                      curexc_traceback
pandas/_libs/algos.c:138983:13: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_type’; did you mean ‘curexc_type’?
     tstate->exc_type = local_type;
             ^~~~~~~~
             curexc_type
pandas/_libs/algos.c:138984:13: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_value’; did you mean ‘curexc_value’?
     tstate->exc_value = local_value;
             ^~~~~~~~~
             curexc_value
pandas/_libs/algos.c:138985:13: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_traceback’; did you mean ‘curexc_traceback’?
     tstate->exc_traceback = local_tb;
             ^~~~~~~~~~~~~
             curexc_traceback
pandas/_libs/algos.c: In function ‘__Pyx_PyCFunction_FastCall’:
pandas/_libs/algos.c:139177:13: error: too many arguments to function ‘(PyObject * (*)(PyObject *, PyObject * const*, Py_ssize_t))meth’
     return (*((__Pyx_PyCFunctionFast)meth)) (self, args, nargs, NULL);
            ~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
pandas/_libs/algos.c: In function ‘__Pyx__ExceptionSwap’:
pandas/_libs/algos.c:141175:24: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_type’; did you mean ‘curexc_type’?
     tmp_type = tstate->exc_type;
                        ^~~~~~~~
                        curexc_type
pandas/_libs/algos.c:141176:25: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_value’; did you mean ‘curexc_value’?
     tmp_value = tstate->exc_value;
                         ^~~~~~~~~
                         curexc_value
pandas/_libs/algos.c:141177:22: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_traceback’; did you mean ‘curexc_traceback’?
     tmp_tb = tstate->exc_traceback;
                      ^~~~~~~~~~~~~
                      curexc_traceback
pandas/_libs/algos.c:141178:13: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_type’; did you mean ‘curexc_type’?
     tstate->exc_type = *type;
             ^~~~~~~~
             curexc_type
pandas/_libs/algos.c:141179:13: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_value’; did you mean ‘curexc_value’?
     tstate->exc_value = *value;
             ^~~~~~~~~
             curexc_value
pandas/_libs/algos.c:141180:13: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_traceback’; did you mean ‘curexc_traceback’?
     tstate->exc_traceback = *tb;
             ^~~~~~~~~~~~~
             curexc_traceback
error: command 'gcc' failed with exit status 1

(for what it's worth, I'm, seeing the same problem when building with either GCC 7.3.0 or Intel compilers 2018 update 3, but I don't think that's relevant giving the produced errors)

numpy has seem similar issues, see numpy/numpy#10500.

It seems like this issue is caused by a known & fixed bug in Cython, cfr. cython/cython#1955.

So the pandas 0.23.2 release includes files that were generated using a too old version of Cython?

I have the latest Cython release installed on top of Python 3.7.0, but the skipping 'pandas/_libs/algos.c' Cython extension (up-to-date) tells me it's not being used?

[TomAugspurger]

So the pandas 0.23.2 release includes files that were generated using a too old version of Cython?

Hmm I'm not quite sure what happened, but it seems like the C files in the sdist were generated by Cython 0.25.2, which is probably too old.

[jorisvandenbossche]

Thanks for the report! We will do a 0.23.3 release to fix this

[boegel]

Excellent, thanks a lot for confirming the problem, and doing the right thing (i.e. no sneaky re-release under the same version number)! 👍

[jorisvandenbossche]

PyPI forces us here to do the right thing :-)

[jorisvandenbossche]

It's released, wheels are building, when they are ready, will be pushed to PyPI together with the sdist