MAINT: Const qualify UFunc inner loops #15355

Kai-Striega · 2020-01-20T11:18:48Z

This PR const qualifies the dimension and strides arguments of PyUFuncGenericFunction. Const qualified arguments make it simpler to reason about the behaviour of these ufuncs and prevents accidental mutation. As the const is now required this PR also const qualifies calls to PyUFuncGenericFunction inside the NumPy source code.

This closes #15252

This PR const qualifies the ``dimension`` and ``strides`` arguments of ``PyUFuncGenericFunction``. Const qualified arguments make it simpler to reason about the behaviour of these ufuncs and prevents accidental mutatation. As the const is now required this PR also const qualifies calls to ``PyUFuncGenericFunction`` inside the NumPy source code.

Kai-Striega · 2020-01-20T11:23:49Z

numpy/__init__.pxd

@@ -841,7 +841,7 @@ cdef inline char* _util_dtypestring(dtype descr, char* f, char* end, int* offset

 cdef extern from "numpy/ufuncobject.h":

-    ctypedef void (*PyUFuncGenericFunction) (char **, npy_intp *, npy_intp *, void *)
+    ctypedef void (*PyUFuncGenericFunction) (char **, const npy_intp *, const npy_intp *, void *)


Here and below I'm using west const ((char **, const npy_intp *, const npy_intp *, void *)). This is different to much of the existing east const code as cython doesn't seem to compile east const expressions. Should this be added as a note/fixme? And should this be added as a Cython issue?

As far as I can tell, we do not include any const in our cython code, so ok to omit this for now.

I would definitely be in favor of filing a feature request against cython to support east const :) (xref https://stackoverflow.com/questions/56947927/102441)

I would definitely be in favor of filing a feature request against cython to support east const :)

It looks like there is an open issue already see cython-1475.

As far as I can tell, we do not include any const in our cython code, so ok to omit this for now.

Ok. I've removed the const qualifiers for now. It looks like Cython added const support in v0.18 with the latest release being 0.28 - I don't know the minimum NumPy requirement.

eric-wieser

Generally looks good. Please push any fixups as separate commits, to make subsequent review easier - we can squash at merge time.

eric-wieser · 2020-01-20T11:48:54Z

Can you remove this now-incorrect comment and unnecessary cast?

numpy/numpy/core/src/umath/ufunc_object.c

Lines 3505 to 3510 in 06a19d6

    
                       /* gh-15252: The signature of the inner loop considers `countptr` 
        
                        * mutable. Inner loops aren't actually allowed to modify this 
        
                        * though, so it's fine to cast it. 
        
                        */ 
        
                       innerloop(dataptrs_copy, (npy_intp *)countptr, 
        
                                 strides_copy, innerloopdata);

Previously the `innerlooop` (a PyUFuncGenericFunction) allowed pointers to non-const qualified dimensions. PyUFuncGenericFunction now only accepts const qualified dimensions making the cast, and explaining comment, obselete.

charris · 2020-01-21T15:59:54Z

I don't know the minimum NumPy requirement.

Cython 0.28.14 for Python 3.8 support.

eric-wieser · 2020-01-21T17:28:52Z

Let's leave cython const correctness to a follow-up

doc/release/upcoming_changes/15355.c_api.rst

Co-Authored-By: Sebastian Berg <[email protected]>

seberg · 2020-01-22T00:48:26Z

Thanks @Kai-Striega and Eric. The cython issue is addressed, the release notes seem fine to me and the changes also look good, so putting this in.

Since `numpy==1.19.0` contains at least one breaking ABI change (numpy/numpy#15355), we need to either upper bound the dependency's range or fix our code to support both ABIs. Since the second requires more changes, we prefer the first one for now. PiperOrigin-RevId: 317699730 Change-Id: Ia62a779f9ec42d63d3fac1b69cd75e6084358d2f

Fix numpy to pre-1.19.0 because of breaking ABI change in numpy 1.19.0 (numpy/numpy#15355) See tensorflow/tensorflow#40688.

Also cleans up `setup.py`, upgrading dependencies, adding comments. Fixes #41902 by changing constraint on `numpy` dependency to require `numpy` above the C++ API breaking change (numpy/numpy#15355, 79518fa) Fixes duplicate #43679 (duplicate of #41902) Tested: ```console (venv) mihaimaruseac@ankh:/tmp/tf/cp/tf$ bazel build //tensorflow/tools/pip_package:build_pip_package ... (venv) mihaimaruseac@ankh:/tmp/tf/cp/tf$ ./bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg ... (venv) mihaimaruseac@ankh:/tmp/tf/cp/tf$ pip install --upgrade pip ... (venv) mihaimaruseac@ankh:/tmp/tf/cp/tf$ pip install --upgrade pip Collecting pip Using cached pip-20.2.3-py2.py3-none-any.whl (1.5 MB) Installing collected packages: pip Attempting uninstall: pip Found existing installation: pip 20.1.1 Uninstalling pip-20.1.1: Successfully uninstalled pip-20.1.1 Successfully installed pip-20.2.3 (venv) mihaimaruseac@ankh:/tmp/tf/cp/tf$ pip --version pip 20.2.3 from /tmp/tf/venv/lib/python3.8/site-packages/pip (python 3.8) (venv) mihaimaruseac@ankh:/tmp/tf/cp/tf$ pip list Package Version ------------------- ------- Keras-Preprocessing 1.1.2 numpy 1.19.2 pip 20.2.3 pkg-resources 0.0.0 setuptools 44.0.0 six 1.15.0 wheel 0.35.1 (venv) mihaimaruseac@ankh:/tmp/tf/cp/tf$ python -m pip install --upgrade --use-feature=2020-resolver /tmp/tensorflow_pkg/tensorflow-2.4.0-cp38-cp38-linux_x86_64.whl ... Successfully installed absl-py-0.10.0 astunparse-1.6.3 cachetools-4.1.1 certifi-2020.6.20 chardet-3.0.4 flatbuffers-1.12 gast-0.4.0 google-auth-1.22.1 google-auth-oauthlib-0.4.1 google-pasta-0.2.0 grpcio-1.32.0 h5py-2.10.0 idna-2.10 markdown-3.3.1 oauthlib-3.1.0 opt-einsum-3.3.0 protobuf-3.13.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 requests-2.24.0 requests-oauthlib-1.3.0 rsa-4.6 tensorboard-2.3.0 tensorboard-plugin-wit-1.7.0 tensorflow-2.4.0 tensorflow-estimator-2.3.0 termcolor-1.1.0 typing-extensions-3.7.4.3 urllib3-1.25.10 werkzeug-1.0.1 wrapt-1.12.1 (venv) mihaimaruseac@ankh:/tmp/tf/cp/tf$ pip list Package Version ---------------------- --------- absl-py 0.10.0 astunparse 1.6.3 cachetools 4.1.1 certifi 2020.6.20 chardet 3.0.4 flatbuffers 1.12 gast 0.4.0 google-auth 1.22.1 google-auth-oauthlib 0.4.1 google-pasta 0.2.0 grpcio 1.32.0 h5py 2.10.0 idna 2.10 Keras-Preprocessing 1.1.2 Markdown 3.3.1 numpy 1.19.2 oauthlib 3.1.0 opt-einsum 3.3.0 pip 20.2.3 pkg-resources 0.0.0 protobuf 3.13.0 pyasn1 0.4.8 pyasn1-modules 0.2.8 requests 2.24.0 requests-oauthlib 1.3.0 rsa 4.6 setuptools 44.0.0 six 1.15.0 tensorboard 2.3.0 tensorboard-plugin-wit 1.7.0 tensorflow 2.4.0 tensorflow-estimator 2.3.0 termcolor 1.1.0 typing-extensions 3.7.4.3 urllib3 1.25.10 Werkzeug 1.0.1 wheel 0.35.1 wrapt 1.12.1 ``` PiperOrigin-RevId: 337563781 Change-Id: I2aa111b21a43e333d6891c0392051d765aade964

rgommers · 2021-02-12T21:13:31Z

I landed on this issue in a roundabout way. It looks like this inadvertently broke the ABI, which affected TensorFlow: tensorflow/tensorflow#41902 (among others).

Comment in current TensorFlow master branch:

    # NOTE: As numpy has releases that break semver guarantees and several other
    # deps depend on numpy without an upper bound, we must install numpy before
    # everything else.
    'numpy ~= 1.19.2',

The ~= means >= 1.19.2; < 1.20.

The ABI break is a done deal now, I'm just commenting on it because it looks like we weren't even aware of it - is that right @seberg, @eric-wieser?

The version pinning thing is something we really should address with some guidance. What TensorFlow does is too restrictive and is going to cause problems if more packages start doing that. But the habit of not specifying any upper bound is indeed a real problem, we should get packages to change that. I'll open a new issue for this.

eric-wieser · 2021-02-12T21:25:26Z

I was completely unaware this was an ABI change. Does the tensorflow issue explain what the ABI issue is?

rgommers · 2021-02-12T21:30:44Z

The one I linked to doesn't, but this may: tensorflow/tensorflow#40728 (comment)

eric-wieser · 2021-02-12T21:34:11Z

My reading of this is that the reviewers were confused, and concluded ultimately that this was an API breakage (which we did know), not an ABI breakage.

seberg · 2021-02-12T21:47:54Z

I don't know, but it sounds like a lot of confusion about "link time", but there is no link time with regard to the NumPy API (since we use an API function table instead). If there was an actual ABI break, SciPy and a whole lot of things would have gone down in flames. (How did SciPy fix this? Ignore the compile warnings?)

We were aware that this might force users to ignore warning or const qualify there loops. I did not realize that you might have to do both to get a warning free compile on either version of NumPy. Or am I misunderstanding some C++ foo?

rgommers · 2021-02-12T21:47:59Z

That discussion is indeed confusing. They seemed to agree on API break in the end. The trouble seems to be more that the impact was underestimated in gh-15252. You briefly said "force users to add const to their loop signatures", but the rest of the discussion was only about compiler warnings, not that this would break already released packages. The issue doesn't even say "this will be an API break".

rgommers · 2021-02-12T21:48:52Z

Maybe it's only a warning in C, but a hard error in C++?

seberg · 2021-02-12T21:49:15Z

@rgommers do you have an example of an already released package breaking? I don't understand how symbol names can matter considering the way our API is exported!

rgommers · 2021-02-12T21:55:36Z

Never mind, I think that was more incorrect info. tensorflow/tensorflow#41902 says that opting in to the new pip resolver, which got people a different numpy version, broke tensorflow. But reading the long linked Pip issue I don't think that's correct - it involved a source build breaking.

seberg · 2021-02-13T00:03:30Z

OK, so the problem is that we were only aware that this will cause compiler warnings in C (which seems acceptable and even good). But, it actually causes C++ to barf, because C++ is much stricter about const (making this a real API break).

As is, I guess a code fix in tensorflow should not have been awfully hard. In some comments it sounds like the reasoning for not doing this (as in a forced cast) was fearing an actual ABI/API break 😞; which simply won't happen.
(What will probably happen at some point is that we deprecate the PyUFunc_RegisterLoopForDescr function through Python warnings.)

In any case, even forcing tensorflow to modify code to compile successfully for both versions was obviously never intended here. It would have been nice to know about the problem at a time when reverting might still have been helpful, we could even have had a 1.19.1 within days of the problem being noticed (and since it was not an ABI break, probably very few would have been hit).

(I don't know C++ well enough to know if there might be some other actual "warning only" solution for C++.)

eric-wieser · 2021-02-13T01:40:42Z

You briefly said "force users to add const to their loop signatures", but the rest of the discussion was only about compiler warnings, not that this would break already released packages.

Yes, I guess I forgot that already-released source (but not binary) packages are affected by API changes when users build them from source against the latest numpy.

See numpy/numpy#15355.

Kai-Striega requested a review from eric-wieser January 20, 2020 11:18

Kai-Striega commented Jan 20, 2020

View reviewed changes

eric-wieser reviewed Jan 20, 2020

View reviewed changes

eric-wieser added 03 - Maintenance 56 - Needs Release Note. Needs an entry in doc/release/upcoming_changes labels Jan 20, 2020

Kai-Striega added 3 commits January 20, 2020 20:51

REVERT: Remove const qualifiers from Cython code

eaeb5b8

MAINT: Remove (now) incorrect comment and cast

328d8fc

Previously the `innerlooop` (a PyUFuncGenericFunction) allowed pointers to non-const qualified dimensions. PyUFuncGenericFunction now only accepts const qualified dimensions making the cast, and explaining comment, obselete.

MAINT: Add release note

a04380b

seberg reviewed Jan 21, 2020

View reviewed changes

doc/release/upcoming_changes/15355.c_api.rst Outdated Show resolved Hide resolved

seberg removed the 56 - Needs Release Note. Needs an entry in doc/release/upcoming_changes label Jan 21, 2020

Use double backticks for code snippet 15355.c_api.rst

a8ec7eb

Co-Authored-By: Sebastian Berg <[email protected]>

seberg merged commit e94cec8 into numpy:master Jan 22, 2020

Kai-Striega deleted the make-ufunc-loop-signature-const-correct branch January 22, 2020 04:55

adk9 mentioned this pull request Jun 23, 2020

C++ compilation of rule '//tensorflow/python:bfloat16_lib' failed tensorflow/tensorflow#40688

Closed

adk9 added a commit to adk9/docs that referenced this pull request Jun 23, 2020

Fix numpy version to pre-1.19.0

1d21b64

Fix numpy to pre-1.19.0 because of breaking ABI change in numpy 1.19.0 (numpy/numpy#15355) See tensorflow/tensorflow#40688.

This was referenced Jun 23, 2020

Fix numpy version to pre-1.19.0 adk9/docs#1

Merged

Fix numpy version to pre-1.19.0 tensorflow/docs#1603

Merged

edudev mentioned this pull request Jun 23, 2020

Const-qulify PyUFuncGenericFunction's arguments tensorflow/tensorflow#40728

Closed

ozon2 mentioned this pull request Jul 6, 2020

Build error with numpy 1.19.0 stack-of-tasks/eigenpy#202

Closed

lpsinger mentioned this pull request Jul 9, 2020

Fix const correctness in Numpy ufunc API healpy/healpy#624

Merged

di mentioned this pull request Jul 30, 2020

Relax constraint on numpy dependency tensorflow/tensorflow#41902

Closed

This was referenced Sep 23, 2020

Do not recommend using TensorFlow<2.3>=2.0 with NumPy>=1.19.0 thoth-station/adviser#1262

Merged

Add justification for tf_41902 thoth-station/thoth-station.github.io#163

Merged

casassg mentioned this pull request Sep 30, 2020

relaxing numpy requirement tensorflow/tensorflow#43679

Closed

rgommers mentioned this pull request Feb 12, 2021

Document version pinning strategy downstream packages should use #18406

Closed

lpsinger mentioned this pull request Nov 11, 2023

Remove old placeholder workaround astropy/astropy-healpix#198

Merged

lpsinger added a commit to lpsinger/astropy-healpix that referenced this pull request Nov 11, 2023

Make Numpy loop function args const correct

ccf6d9e

See numpy/numpy#15355.

lpsinger mentioned this pull request Nov 11, 2023

Make Numpy loop function args const correct astropy/astropy-healpix#199

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MAINT: Const qualify UFunc inner loops #15355

MAINT: Const qualify UFunc inner loops #15355

Kai-Striega commented Jan 20, 2020

Kai-Striega Jan 20, 2020

eric-wieser Jan 20, 2020 •

edited

Loading

Kai-Striega Jan 21, 2020

eric-wieser left a comment

eric-wieser commented Jan 20, 2020 •

edited

Loading

charris commented Jan 21, 2020

eric-wieser commented Jan 21, 2020

seberg commented Jan 22, 2020

rgommers commented Feb 12, 2021 •

edited

Loading

eric-wieser commented Feb 12, 2021

rgommers commented Feb 12, 2021

eric-wieser commented Feb 12, 2021

seberg commented Feb 12, 2021

rgommers commented Feb 12, 2021

rgommers commented Feb 12, 2021

seberg commented Feb 12, 2021

rgommers commented Feb 12, 2021

seberg commented Feb 13, 2021

eric-wieser commented Feb 13, 2021

MAINT: Const qualify UFunc inner loops #15355

MAINT: Const qualify UFunc inner loops #15355

Conversation

Kai-Striega commented Jan 20, 2020

Kai-Striega Jan 20, 2020

Choose a reason for hiding this comment

eric-wieser Jan 20, 2020 • edited Loading

Choose a reason for hiding this comment

Kai-Striega Jan 21, 2020

Choose a reason for hiding this comment

eric-wieser left a comment

Choose a reason for hiding this comment

eric-wieser commented Jan 20, 2020 • edited Loading

charris commented Jan 21, 2020

eric-wieser commented Jan 21, 2020

seberg commented Jan 22, 2020

rgommers commented Feb 12, 2021 • edited Loading

eric-wieser commented Feb 12, 2021

rgommers commented Feb 12, 2021

eric-wieser commented Feb 12, 2021

seberg commented Feb 12, 2021

rgommers commented Feb 12, 2021

rgommers commented Feb 12, 2021

seberg commented Feb 12, 2021

rgommers commented Feb 12, 2021

seberg commented Feb 13, 2021

eric-wieser commented Feb 13, 2021

eric-wieser Jan 20, 2020 •

edited

Loading

eric-wieser commented Jan 20, 2020 •

edited

Loading

rgommers commented Feb 12, 2021 •

edited

Loading