API: bump MAXDIMS/MAXARGS to 64 introduce NPY_AXIS_RAVEL #25149

seberg · 2023-11-14T21:42:04Z

Bumping these two has three main caveats:

We cannot break ABI for the iterator macros, which means the
iterators now need to use 32/their own macro.
We used axis=MAXDIMS to mean axis=None, introduce NPY_AXIS_RAVEL
to replace this, it will be run-time outside NumPy.
The old style iterators cannot deal with high dimensions, meaning
that some functions just won't work (because they use them).

Draft, as based on the mapiter removal. ~~Needs to document the above points, but~~ should be good for an initial review ~~(the doc changes here may be a bit random). But,~~ I don't expect the code to change really.

There are two plausible follow-ups that I would prefer not to include here:

I had some start to use alloca() or similar for some allocations (or malloc for big ones). We don't have to do go all the way there, but at least the stack arrays in the ufunc setup are worth shrinking. (definitely want to do that)
It might be nice to address some of the places that remain limited. Examples are the choose and arr.flat I added test for here. As well as np.random since it relies on the old-style iterator a lot.
- I don't have a real solution here, we could duplicate the old style iterators, giving you more dimensions (if you rely on NumPy >2, backporting isn't plausible).

I had mentioned before to just delete NPY_MAXDIMS as a public symbol rather than deleting it. After seeing that it is used in a few places in SciPy, etc. I decided that maybe it is best to defer that to NumPy 3. I doubt we are ready to force that churn on downstream.

Addresses gh-5744, although some paths remain of course.

seberg · 2023-11-15T11:01:37Z

OK, I think this is now ready (but relying on the other PR). I had not done it here, but I would be happy to follow up with:

Removing NPY_MAXARGS. The main useful "public" use seems at the end of the iterator structs (and it is OK to change it there). (could also do this here, but not sure it helps)
I checked and Cython does not define arr.flat to return a C iterator. So I think we can make arr.flat return a new object that supports more dimensions.
This may not be a super quick thing, unfortunately. Unless we mostly copy paste the old one. The reason is that I am not sure the indexing code can be redone very easily. (Even if it would be nice anyway)

mattip · 2023-11-15T11:14:48Z

the other PR

Link?

seberg · 2023-11-15T11:20:22Z

Oops, thought it was in the first post, just opened: #25138

seberg · 2023-11-15T15:47:52Z

numpy/_core/src/multiarray/conversion_utils.c

-                          "(Deprecated NumPy 1.23)") < 0) {
-                return NPY_FAIL;
-            }
-        }


This is very niche, but, mentioned it in the release notes.

seberg · 2023-11-15T15:48:33Z

numpy/_core/src/multiarray/multiarraymodule.c

@@ -692,7 +692,7 @@ PyArray_ConcatenateInto(PyObject *op,
        Py_DECREF(item);
    }

-    if (axis >= NPY_MAXDIMS) {
+    if (axis == NPY_RAVEL_AXIS) {


Also a subtle oddity, mentioned in the release note (any huge axes was interpreted as raveling)

seberg · 2023-11-15T15:50:01Z

numpy/_core/src/multiarray/refcount.c

+                    "this function only supports up to 32 dimensions but "
+                    "the array has %d.", PyArray_NDIM(mp));
+            return -1;
+        }


We don't us this internally, might remove the function, but not sure yet. In principle, anyone running into it should catch the error and not leak anything (normal array cleanup doesn't have this problem).

seberg · 2023-11-15T16:10:27Z

numpy/__init__.pxd

@@ -182,9 +182,8 @@ cdef extern from "numpy/arrayobject.h":
        NPY_ARRAY_UPDATE_ALL

    cdef enum:
-        NPY_MAXDIMS
-
-    npy_intp NPY_MAX_ELSIZE


Maybe a bit random, but NPY_MAX_ELSIZE doesn't exist (it sure did at some point).

Does this removal need a release note? Just to say that if someone was using and relying on it, it wasn't doing what they thought it did.

🤷 I doubt anyone will notice, but sure, added.

Bumping these two has three main caveats: 1. We cannot break ABI for the iterator macros, which means the iterators now need to use 32/their own macro. 2. We used axis=MAXDIMS to mean axis=None, introduce NPY_AXIS_RAVEL to replace this, it will be run-time outside NumPy. 3. The old style iterators cannot deal with high dimensions, meaning that some functions just won't work (because they use them).

…ader

Memoryview actually also only supports 65 dimensions currently, so skip the test when the memoryview creation fails. If it were to work, the test should kick in and work fine.

seberg · 2023-11-22T14:13:47Z

@ngoldbaum maybe you have some time to review this?

ngoldbaum

A few language suggestions and some questions. Overall looks fine though.

doc/source/numpy_2_0_migration_guide.rst

ngoldbaum · 2023-11-22T16:15:55Z

doc/release/upcoming_changes/25149.c_api.rst

+used in stack allocation, where the increase should be safe.
+However, we do encourage generally to remove any use of ``NPY_MAXDIMS`` and
+``NPY_MAXARGS`` to eventually allow removing the constraint completely.
+In some cases, ``NPY_MAXDIMS`` was passed (and returned) to mean ``axis=None``


Can you make "In some cases" a little more specific? Does this only impact things that interact with numpy via the C API, or could you trigger this behavior from the python API too?

The python side places are already mentioned in the other one, but made it a bit more concrete.

doc/release/upcoming_changes/25149.c_api.rst

ngoldbaum · 2023-11-22T16:16:50Z

doc/release/upcoming_changes/25149.change.rst

+-----------------------------------------------
+In some cases ``axis=32`` or for concatenate any large value
+was the same as ``axis=None``.
+Except for ``concatenate`` this was deprecate.


Suggested change

Except for ``concatenate`` this was deprecate.

Except for ``concatenate`` this was deprecated in Numpy 1.23.

Was the deprecation noisy for other functions? Just wondering if we might want to merely deprecate this usage for concatenate since that was missed in 2022.

It gave a deprecation warning. I briefly considered that, and think that this is ridiculously niche. IIRC, we were close to not do the deprecation warning in the other path also.

We are talking about np.concatenate(..., axis=32) or higher. Where the axis value cannot relate to any actual axis.
This is undocumented and unless you do it accidentally (and it happens to do what you want?!) nobody should have reason to even realize it works (at least for ~15 years), I would say.

ngoldbaum · 2023-11-22T16:17:35Z

doc/source/numpy_2_0_migration_guide.rst

+``NPY_MAXDIMS`` was also used to signal ``axis=None`` in the C-API, including
+the ``PyArray_AxisConverter``.
+If you run into this problem, you will see ``-2147483648``
+(the minimum integer value) or ``64`` being used as an invalid axis.


It might help to include the exact text of the error they'd see, so if they google the error message they'll find the migration guide

doc/source/reference/c-api/array.rst

ngoldbaum · 2023-11-22T16:23:55Z

doc/source/reference/c-api/types-and-structures.rst

-       is the largest number of dimensions for any array.
+       ``ndarraytypes.h`` points to this data member.
+       Although most operations may be limited in dimensionality, we do not
+       advertise a maximum dimension.  Anyone explicitly relying on one


Is this a remnant of an earlier version that completely removed NPY_MAXDIMS from the public API? Maybe rephrase this to say NPY_MAXDIMS is there but not to rely on it because it might change or be removed in the future.

It wasn't a remnant, but rephrased a bit.

ngoldbaum · 2023-11-22T16:25:30Z

numpy/__init__.pxd

@@ -182,9 +182,8 @@ cdef extern from "numpy/arrayobject.h":
        NPY_ARRAY_UPDATE_ALL

    cdef enum:
-        NPY_MAXDIMS
-
-    npy_intp NPY_MAX_ELSIZE


Does this removal need a release note? Just to say that if someone was using and relying on it, it wasn't doing what they thought it did.

Co-authored-by: Nathan Goldbaum <[email protected]>

ngoldbaum

I'm on vacation but will merge this next week if no one else does it before me. This is mostly very straightforward and it's hard to know the impact of the ABI break or possible breakage of questionable C code using the macro without making the change.

ngoldbaum · 2023-11-28T19:16:20Z

I'm going to merge this now and immediately trigger a new nightly wheel upload. Thanks @seberg!

If you are reading this because of new CI breakage, make sure any nightly builds or tests you're running against NumPy dev (future 2.0 release) use a stack that is built against Numpy dev as well, with build isolation disabled.

github-actions bot added the 30 - API label Nov 14, 2023

seberg marked this pull request as draft November 14, 2023 21:42

seberg force-pushed the maxdims branch from 0a6dbd1 to 3026cd9 Compare November 15, 2023 09:59

seberg force-pushed the maxdims branch from 3a365b6 to 4ab9ce9 Compare November 15, 2023 14:45

seberg marked this pull request as ready for review November 15, 2023 14:45

seberg commented Nov 15, 2023

View reviewed changes

seberg force-pushed the maxdims branch from 71d6b63 to afc421d Compare November 15, 2023 16:05

seberg commented Nov 15, 2023

View reviewed changes

seberg changed the title ~~API: NumPy NPY_MAXDIMS and NPY_MAXARGS to 64 introduce NPY_AXIS_RAVEL~~ API: bump NPY_MAXDIMS and NPY_MAXARGS to 64 introduce NPY_AXIS_RAVEL Nov 16, 2023

seberg changed the title ~~API: bump NPY_MAXDIMS and NPY_MAXARGS to 64 introduce NPY_AXIS_RAVEL~~ API: bump MAXDIMS/MAXARGS to 64 introduce NPY_AXIS_RAVEL Nov 16, 2023

seberg added 8 commits November 17, 2023 15:47

TST: Adapt test to larger max-dims and max-args

af55c6b

TST: Add a few tests for paths that are for now limited to 32 dims

bbc4083

API: Add 1.x support for PyArray_RUNTIME_VERSION to the compat he…

6a878e3

…ader

DOC: Add documentation for maxdim increase (and related) changes

3270231

TST: "fix" failing memoryview with too many dimensions test

d6840f5

Memoryview actually also only supports 65 dimensions currently, so skip the test when the memoryview creation fails. If it were to work, the test should kick in and work fine.

TST: Make linter happy, it was right.

a5a250f

MAINT: Cython restore NPY_MAXDIMS and add NPY_RAVEL_AXIS

8dddc54

seberg force-pushed the maxdims branch from afc421d to 8dddc54 Compare November 17, 2023 14:48

ngoldbaum reviewed Nov 22, 2023

View reviewed changes

seberg and others added 2 commits November 24, 2023 12:20

Apply suggestions from code review

56b792c

Co-authored-by: Nathan Goldbaum <[email protected]>

Address review comments

0498d8c

ngoldbaum approved these changes Nov 24, 2023

View reviewed changes

ngoldbaum merged commit a5b0831 into numpy:main Nov 28, 2023

seberg deleted the maxdims branch November 29, 2023 08:17

This was referenced Dec 6, 2023

BUG: fix refcounting for dtypemeta aliases #25329

Merged

BUG: pypy39 wheel builder failing #25334

Closed

rgommers mentioned this pull request Jun 24, 2024

Increase NPY_MAXARGS to more than 32 #4398

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

API: bump MAXDIMS/MAXARGS to 64 introduce NPY_AXIS_RAVEL #25149

API: bump MAXDIMS/MAXARGS to 64 introduce NPY_AXIS_RAVEL #25149

seberg commented Nov 14, 2023 •

edited

Loading

seberg commented Nov 15, 2023

mattip commented Nov 15, 2023

seberg commented Nov 15, 2023

seberg Nov 15, 2023

seberg Nov 15, 2023

seberg Nov 15, 2023

seberg Nov 15, 2023

ngoldbaum Nov 22, 2023

seberg Nov 24, 2023

seberg commented Nov 22, 2023

ngoldbaum left a comment

ngoldbaum Nov 22, 2023

seberg Nov 24, 2023

ngoldbaum Nov 22, 2023

seberg Nov 24, 2023

ngoldbaum Nov 22, 2023

ngoldbaum Nov 22, 2023

seberg Nov 24, 2023

ngoldbaum Nov 22, 2023

ngoldbaum left a comment

ngoldbaum commented Nov 28, 2023

	Except for ``concatenate`` this was deprecate.
	Except for ``concatenate`` this was deprecated in Numpy 1.23.

API: bump MAXDIMS/MAXARGS to 64 introduce NPY_AXIS_RAVEL #25149

API: bump MAXDIMS/MAXARGS to 64 introduce NPY_AXIS_RAVEL #25149

Conversation

seberg commented Nov 14, 2023 • edited Loading

seberg commented Nov 15, 2023

mattip commented Nov 15, 2023

seberg commented Nov 15, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

seberg commented Nov 22, 2023

ngoldbaum left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ngoldbaum left a comment

Choose a reason for hiding this comment

ngoldbaum commented Nov 28, 2023

seberg commented Nov 14, 2023 •

edited

Loading