BUG: DataFrameGroupBy.transform with axis=1 fails (#36308)

[ethanator]

• closes BUG: DataFrameGroupBy.transform with axis=1 fails #36308
• tests added / passed
• passes black pandas
• passes git diff upstream/master -u -- "*.py" | flake8 --diff
• whatsnew entry

Updated pandas.core.groupby.generic._wrap_transformed_output as suggested by @rhshadrach but also had to modify pandas.core.groupby.groupby.pct_change to get 2 tests passed where the operation passed to transform is pct_change.

All tests passed except for pandas/tests/io/test_clipboard.py and pandas/tests/io/test_parquet.py that are already broken on the latest master as of September 14, 2020 01:15 UTC.

[dsaxton]

Thanks @ethanator, can you also add a test and bug fix note in the v1.1.3 whatsnew?

[jreback]

put this on 1.2, pls add a test for shift and for pct_change which was modified as well.

[rhshadrach]

@ethanator Are you still interested in working on this?

[ethanator]

@ethanator Are you still interested in working on this?

Yes. I got distracted but I'm adding tests now to wrap this up.

[github-actions]

This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.

[rhshadrach]

@ethanator I'd like to get this in 1.2, would it be okay if I finished it up?

[jbrockmendel]

I'd like to get this in 1.2, would it be okay if I finished it up?

yes, if you can address comments about tests and merge master this looked almost ready

[rhshadrach]

Both shift and pct_change are included in the transformation functions in the added test.

[rhshadrach]

Test fails on Windows 32-bit because DataFrame gets constructed with 32-bit ints but doing df["b"] = df["b"].astype(int) results in 64-bit ints. Not sure what a good way to do this is; maybe specify dtype=int in the DataFrame construction?

[jbrockmendel]

Well which is the "correct" answer? if you want to get int32 when on a 32bit machine you can do

expected["b"] = expected["b"].astype(np.intp)

[rhshadrach]

Thanks @jbrockmendel, that's exactly what I was asking but it turns out I had it backwards. The result in the test was 64-bit, but using astype(int) on expected (to undo the coercion to float in the expected computation) resulted in 32-bit. I've updated the test to use astype("int64").

However, I'm not sure if this is really the correct behavior. Should the result be coming back as 64-bit here as is currently the case?

[jreback]

Thanks @jbrockmendel, that's exactly what I was asking but it turns out I had it backwards. The result in the test was 64-bit, but using astype(int) on expected (to undo the coercion to float in the expected computation) resulted in 32-bit. I've updated the test to use astype("int64").

However, I'm not sure if this is really the correct behavior. Should the result be coming back as 64-bit here as is currently the case?

your inputs are all int64 (on windows as well), so it IS suprising that you are getting an intp back (32-bit on windows). though i suppose somewhere we are constructing a return array for the ints that uses intp. I would say ok for now, but let's open an issue to dig depper on this.

@rhshadrach ok on this otherwise? (lgtm.).

[rhshadrach]

your inputs are all int64 (on windows as well), so it IS suprising that you are getting an intp back (32-bit on windows).

result in the test is coming back as int64; so I think this is then okay. Unless you're surprised that astype(int) is giving back 32-bit ints on 32-bit Windows? In the call to astype(int), the inputs are floats.

Aside from this, the PR is good on my end.

[jreback]

oh i c you are doing this because we want to compare to ints, got it ,then this is fine.

[jreback]

thanks @ethanator