BUG: Unable to build cython when using NpyIter_GetIterNext.

[zoj613]

Describe the issue:

I am trying to migrate away from using custom cython imports in order to have access to the Iterator API declarations at numpy/ndarrayobject.h by using the numpy provided ones added in v2.0.0. However if I try to build the project using the file here I get a build failure. Cc @seberg

Reproduce the code example:

pip install -r requirements-dev.txt
	cythonize polyagamma/*.pyx
	pip install -e .

Error message:

gcc -pthread -B /home/zoj/.local/share/micromamba/envs/polyagamma-dev/compiler_compat -fno-s
trict-overflow -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /home/zoj/.local/share/micromamba/envs/polyag
amma-dev/include -fPIC -O2 -isystem /home/zoj/.local/share/micromamba/envs/polyagamma-dev/include 
-fPIC -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -I/tmp/pip-build-env-0bay5vae/overlay/lib/python
3.12/site-packages/numpy/_core/include -I./include -I/home/zoj/.local/share/micromamba/envs/polyag
amma-dev/include/python3.12 -c polyagamma/_polyagamma.c -o /tmp/tmpsbownh73.build-temp/polyagamma/
_polyagamma.o -O2 -std=c99
      polyagamma/_polyagamma.c: In function ‘__pyx_pf_10polyagamma_11_polyagamma_rvs’:
      polyagamma/_polyagamma.c:6265:16: error: assignment to ‘int (**)(NpyIter *)’ {aka ‘int (**)(
struct NpyIter_InternalOnly *)’} from incompatible pointer type ‘int (*)(NpyIter *)’ {aka ‘int (*)
(struct NpyIter_InternalOnly *)’} [-Wincompatible-pointer-types]
       6265 |     __pyx_t_20 = NpyIter_GetIterNext(__pyx_v_it, NULL); if (unlikely(__pyx_t_20 == (
(__pyx_t_5numpy_NpyIter_IterNextFunc *)NULL))) __PYX_ERR(0, 243, __pyx_L57_error)
            |                ^
      polyagamma/_polyagamma.c: In function ‘__pyx_f_10polyagamma_11_polyagamma_dispatch’:
      polyagamma/_polyagamma.c:7507:16: error: assignment to ‘int (**)(NpyIter *)’ {aka ‘int (**)(
struct NpyIter_InternalOnly *)’} from incompatible pointer type ‘int (*)(NpyIter *)’ {aka ‘int (*)
(struct NpyIter_InternalOnly *)’} [-Wincompatible-pointer-types]
       7507 |     __pyx_t_10 = NpyIter_GetIterNext(__pyx_v_it, NULL); if (unlikely(__pyx_t_10 == (
(__pyx_t_5numpy_NpyIter_IterNextFunc *)NULL))) __PYX_ERR(0, 462, __pyx_L11_error)
            |                ^
      Traceback (most recent call last):
        File "/tmp/pip-build-env-0bay5vae/overlay/lib/python3.12/site-packages/setuptools/_distuti
ls/spawn.py", line 70, in spawn
          subprocess.check_call(cmd, env=_inject_macos_ver(env))
        File "/home/zoj/.local/share/micromamba/envs/polyagamma-dev/lib/python3.12/subprocess.py",
 line 413, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['/usr/bin/gcc', '-pthread', '-B', '/home/zoj/.local
/share/micromamba/envs/polyagamma-dev/compiler_compat', '-fno-strict-overflow', '-DNDEBUG', '-O2',
 '-Wall', '-fPIC', '-O2', '-isystem', '/home/zoj/.local/share/micromamba/envs/polyagamma-dev/inclu
de', '-fPIC', '-O2', '-isystem', '/home/zoj/.local/share/micromamba/envs/polyagamma-dev/include', 
'-fPIC', '-DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION', '-I/tmp/pip-build-env-0bay5vae/overlay/lib
/python3.12/site-packages/numpy/_core/include', '-I./include', '-I/home/zoj/.local/share/micromamb
a/envs/polyagamma-dev/include/python3.12', '-c', 'polyagamma/_polyagamma.c', '-o', '/tmp/tmpsbownh
73.build-temp/polyagamma/_polyagamma.o', '-O2', '-std=c99']' returned non-zero exit status 1.
      
      The above exception was the direct cause of the following exception:
      
      Traceback (most recent call last):
        File "/tmp/pip-build-env-0bay5vae/overlay/lib/python3.12/site-packages/setuptools/_distuti
ls/unixccompiler.py", line 200, in _compile
          self.spawn(compiler_so + cc_args + [src, '-o', obj] + extra_postargs)
        File "/tmp/pip-build-env-0bay5vae/overlay/lib/python3.12/site-packages/setuptools/_distuti
ls/ccompiler.py", line 1052, in spawn
          spawn(cmd, dry_run=self.dry_run, **kwargs)
        File "/tmp/pip-build-env-0bay5vae/overlay/lib/python3.12/site-packages/setuptools/_distuti
ls/spawn.py", line 76, in spawn
          raise DistutilsExecError(
      distutils.errors.DistutilsExecError: command '/usr/bin/gcc' failed with exit code 1
      
      During handling of the above exception, another exception occurred:
      
      Traceback (most recent call last):
        File "/tmp/pip-build-env-0bay5vae/overlay/lib/python3.12/site-packages/setuptools/command/
editable_wheel.py", line 139, in run
          self._create_wheel_file(bdist_wheel)
        File "/tmp/pip-build-env-0bay5vae/overlay/lib/python3.12/site-packages/setuptools/command/
editable_wheel.py", line 340, in _create_wheel_file
          files, mapping = self._run_build_commands(dist_name, unpacked, lib, tmp)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-0bay5vae/overlay/lib/python3.12/site-packages/setuptools/command/
editable_wheel.py", line 263, in _run_build_commands
          self._run_build_subcommands()
        File "/tmp/pip-build-env-0bay5vae/overlay/lib/python3.12/site-packages/setuptools/command/
editable_wheel.py", line 290, in _run_build_subcommands
          self.run_command(name)
        File "/tmp/pip-build-env-0bay5vae/overlay/lib/python3.12/site-packages/setuptools/_distuti
ls/cmd.py", line 339, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-0bay5vae/overlay/lib/python3.12/site-packages/setuptools/dist.py"
, line 999, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-0bay5vae/overlay/lib/python3.12/site-packages/setuptools/_distuti
ls/dist.py", line 1002, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-0bay5vae/overlay/lib/python3.12/site-packages/setuptools/command/
build_ext.py", line 99, in run
          _build_ext.run(self)
        File "/tmp/pip-build-env-0bay5vae/overlay/lib/python3.12/site-packages/setuptools/_distuti
ls/command/build_ext.py", line 365, in run
          self.build_extensions()
        File "/tmp/pip-build-env-0bay5vae/overlay/lib/python3.12/site-packages/setuptools/_distuti
ls/command/build_ext.py", line 481, in build_extensions
          self._build_extensions_serial()
        File "/tmp/pip-build-env-0bay5vae/overlay/lib/python3.12/site-packages/setuptools/_distuti
ls/command/build_ext.py", line 507, in _build_extensions_serial
          self.build_extension(ext)
        File "/tmp/pip-build-env-0bay5vae/overlay/lib/python3.12/site-packages/setuptools/command/
build_ext.py", line 264, in build_extension
          _build_ext.build_extension(self, ext)
        File "/tmp/pip-build-env-0bay5vae/overlay/lib/python3.12/site-packages/setuptools/_distuti
ls/command/build_ext.py", line 562, in build_extension
          objects = self.compiler.compile(
                    ^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-0bay5vae/overlay/lib/python3.12/site-packages/setuptools/_distuti
ls/ccompiler.py", line 607, in compile
          self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
        File "/tmp/pip-build-env-0bay5vae/overlay/lib/python3.12/site-packages/setuptools/_distuti
ls/unixccompiler.py", line 202, in _compile
          raise CompileError(msg)
      distutils.errors.CompileError: command '/usr/bin/gcc' failed with exit code 1
      /tmp/pip-build-env-0bay5vae/overlay/lib/python3.12/site-packages/setuptools/_distutils/dist.
py:1002: _DebuggingTips: Problem in editable installation.
      !!
      
              ********************************************************************************
              An error happened while installing `polyagamma` in editable mode.
      
              The following steps are recommended to help debug this problem:
      
              - Try to install the project normally, without using the editable mode.
                Does the error still persist?
                (If it does, try fixing the problem before attempting the editable mode).
              - If you are using binary extensions, make sure you have all OS-level
                dependencies installed (e.g. compilers, toolchains, binary libraries, ...).
              - Try the latest version of setuptools (maybe the error was already fixed).
              - If you (or your project dependencies) are using any setuptools extension
                or customization, make sure they support the editable mode.
      
              After following the steps above, if the problem still persists and
              you think this is related to how setuptools handles editable installations,
              please submit a reproducible example
              (see https://stackoverflow.com/help/minimal-reproducible-example) to:
      
                  https://github.com/pypa/setuptools/issues
      
              See https://setuptools.pypa.io/en/latest/userguide/development_mode.html for details
.
              ********************************************************************************
      
      !!
        cmd_obj.run()
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building editable for polyagamma
Failed to build polyagamma
ERROR: Could not build wheels for polyagamma, which is required to install pyproject.toml-based pr
ojects
make: *** [Makefile:14: dev] Error 1

Python and NumPy Versions:

3.12.0 | packaged by conda-forge | (main, Oct 3 2023, 08:43:22) [GCC 12.3.0]

Runtime Environment:

[{'numpy_version': '2.2.3',
  'python': '3.12.0 | packaged by conda-forge | (main, Oct  3 2023, 08:43:22) '
            '[GCC 12.3.0]',
  'uname': uname_result(system='Linux', node='zoj-desktop', release='6.12.17-1-lts', version='#1 S
MP PREEMPT_DYNAMIC Thu, 27 Feb 2025 14:12:30 +0000', machine='x86_64')},
 {'simd_extensions': {'baseline': ['SSE', 'SSE2', 'SSE3'],
                      'found': ['SSSE3',
                                'SSE41',
                                'POPCNT',
                                'SSE42',
                                'AVX',
                                'F16C',
                                'FMA3',
                                'AVX2',
                                'AVX512F',
                                'AVX512CD',
                                'AVX512_SKX',
                                'AVX512_CLX',
                                'AVX512_CNL',
                                'AVX512_ICL'],
                      'not_found': ['AVX512_KNL', 'AVX512_KNM']}},
 {'architecture': 'SkylakeX',
  'filepath': '/home/zoj/.local/share/micromamba/envs/polyagamma-dev/lib/python3.12/site-packages/
numpy.libs/libscipy_openblas64_-6bb31eeb.so',
  'internal_api': 'openblas',
  'num_threads': 12,
  'prefix': 'libscipy_openblas',
  'threading_layer': 'pthreads',
  'user_api': 'blas',
  'version': '0.3.28'}]

Context for the issue:

Previously, I was manually importing the cython declarations and defined them like so:

ctypedef int (*NpyIter_IterNextFunc)(NpyIter* it) noexcept nogil
cdef extern from "numpy/ndarrayobject.h":
 NpyIter_IterNextFunc NpyIter_GetIterNext(NpyIter* it, char** errmsg) except NULL
 ...

as shown here. This worked well and I was able to successfully build the project. Now if I use the declarions added in numpy v2.0.0

ctypedef int (*NpyIter_IterNextFunc)(NpyIter* it) noexcept nogil
 NpyIter_IterNextFunc* NpyIter_GetIterNext(NpyIter* it, char** errmsg) except NULL

I get this build error. I believe this is due to NpyIter_GetIterNext returning a pointer to NpyIter_IterNextFunc instead of just the type.

[seberg]

Yeah, seems off... What was the winning combination here? You added this back then and we changed nothing since then, I think.

And I assumed you had code that successfully used it at the time?

I.e. I am worried that we already have someone manages to use this successfully with the current version, and a change may break them.
If that is not possible, great. Can you make a PR to change it, including test coverage for this function which was seems missing in the original PR (it does have a bunch of tests though).

Even if there is a weird way it works, we probably should just change it with a release note asking to either pin at build time, or just copy paste the definition as a work-around.

[seberg]

Hmm, from the PR, maybe it really just needs to not use IterNextFunc *? But, that seems like a difference to C maybe (i.e. the cython versioni includes the pointer, C doesn't). I am not sure if Cython might want that.