Allow aggregate funcs to return arrays in groupby #3805

jtratner · 2013-06-07T23:53:53Z

Please check out whether you like the error message for Performance Warning.

Also, I'm not sure whether this means that groupby fails under certain conditions and not others (like when trying Cython, etc.).

hayd · 2013-06-08T00:28:53Z

Thanks for putting that together. But, do you think we could:

warnings.warn("Function does not produce aggregated values. Will not be able to optimize.", PerformanceWarning)

rather than raise?

hayd · 2013-06-08T00:30:47Z

Hmmm, about the message, we could cover ourselves a bit more and say "Function does not produce aggregated values. Will not be able to optimize and may produce unexpected results."

jtratner · 2013-06-08T00:32:01Z

What's the difference between raise and warnings.warn?

hayd · 2013-06-08T00:33:24Z

~~Oh, I thought that~~ warnings.warn would give you a warning message but still let you do it. Raise kills it, raises an Exception, throws toys out of pram and gives no result. (I could be wrong... but I was hoping for the former :) )

jtratner · 2013-06-08T00:41:57Z

@hayd maybe? I don't know. How do you test that the warning occurred? Also, how about we do even more couched:

Function does not produce aggregated values. Will not be able to optimize and may produce unexpected results. This is experimental and results are not guaranteed to stay the same between versions.

hayd · 2013-06-08T01:21:17Z

I think like this (or do you have it working?).

There are some PerformanceWarnings in test_pytables, but @jreback seems to ignore (?) them with:

warnings.filterwarnings('ignore', category=PerformanceWarning)

Also, I think we may as well include tests for the current results of those calculations. :)

jtratner · 2013-06-08T01:23:03Z

I did add some tests like that with warnings.catch_warning. I'll try and
come back to this tomorrow.

hayd · 2013-06-08T01:24:48Z

(haha! I think the experimental bit... is a little OTT!)

The usual assertEqual(result, expected) tests :)

jtratner · 2013-06-08T01:29:15Z

Given there are multiple interpretations of what it means to produce an
aggregate that's a sequence, I didn't want us to be stuck supporting this
edge case in the future. Your call.

On Fri, Jun 7, 2013 at 9:25 PM, Andy Hayden [email protected]:

(haha! I think the experimental bit... is a little OTT!)

The usual assertEqual(result, expected) tests :)

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/3805#issuecomment-19140771
.

jreback · 2013-06-08T01:30:09Z

@hayd I ignore them so they won't print on the test display; if you don't silence they will...

jtratner · 2013-06-08T01:32:17Z

Or if you use warnings.catch_warnings, you can change the warnings just
for the duration of the test.

hayd · 2013-06-08T01:33:07Z

Ah of course! ~~Do you test the warnings appear?~~

Personally I think that saying "may produce unexpected results" is enough to say we're not really supporting it, and having those assertEquals tests would just highlight when there's a problem and we can then choose to break. :)

jreback · 2013-06-08T01:33:22Z

@jtratner that is the right way......I was ignoring them before I knew how to do that!

jtratner · 2013-06-08T01:34:43Z

Yeah, I just need to fix the tests slightly.

jtratner · 2013-06-08T01:38:07Z

Actually, I'd say that in general we should probably set the warning filter to error so that all errors are explicitly caught (right now Travis has two DeprecationWarning errors).

jreback · 2013-06-08T01:41:38Z

The issue is that some of these warnings we are explicity testing something in order to generate the warning (like here, the Performance issue).....so you would have to have a try except block around them which would mask an actual problem....

jtratner · 2013-06-08T01:47:25Z

Well, if you're testing something to generate the warning, you can do a few things:

Set to 'always' and then assert that the length of the catch_warnings object is greater than 1
Use 'error' and assert that it causes the warning.

I guess 1 is actually better in general.

jtratner · 2013-06-08T01:53:04Z

Also, @hayd I went with your warning message :)

jtratner · 2013-06-08T18:09:18Z

@hayd @jreback does this look okay to you? Do you want me to make any other changes or updates?

hayd · 2013-06-08T18:25:27Z

The only thing I would mention is that we might as well add tests for the actual results of those two calls.

jtratner · 2013-06-08T18:36:30Z

oh sorry! totally forgot. I'll add that.

On Sat, Jun 8, 2013 at 2:25 PM, Andy Hayden [email protected]:

The only thing I would mention is that we might as well add tests for the
actual results of those two calls.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/3805#issuecomment-19153076
.

hayd · 2013-06-08T19:50:21Z

Hmm an additional test to add is the example from the original issue, as I think that is a non ambiguous thing (and after all, it works fine with lists...).

Just going through them, none of the test returned ValueError('Function does not reduce'), can we think of a good/bad example of this, either that's bad or good? (Did you work out why that one was there?)

grouped.agg(lambda x: x.describe())
Exception: Must produce aggregated value
Intermediary output:
count    3.000000
mean     0.971312
std      1.130488
min     -0.330182
25%      0.602385
50%      1.534953
75%      1.622060
max      1.709167
dtype: float64
Now:
A
bar    [3.0, -0.704089998761, 0.619650989953, -1.0771...
foo    [5.0, 0.696226374171, 0.679933321298, 0.288482...
Name: C, dtype: object

grouped.agg(lambda x: x.index[:2])
Exception: Must produce aggregated value
Intermediary output:
Int64Index([1, 3], dtype=int64)
Now:
A
bar    [1, 3]
foo    [0, 2]
Name: C, dtype: object

grouped.aggregate, lambda x: x * 2
Exception: Must produce aggregated value
Intermediary output:
1    3.069905
3   -0.660364
5    3.418333
Name: bar, dtype: float64
Now:
A
bar    [-2.15438895675, -2.09254823116, 0.0223971953452]
foo    [3.78051447736, 0.576965254291, 0.687811502056...
Name: C, dtype: object

In the first example they "obviously" really want the result (and this would also work for the second in this instance):

pd.DataFrame([s for s in res], index=res.index)

I really have no idea what's going on in the third one... unexpected is correct lol.

All cases there is a better way I think.

I see now what you are saying about ambiguity. So maybe tests for these weird example aren't so important... :S

jtratner · 2013-06-08T20:02:42Z

Yeah, and it makes the test cases really annoying, because they have to deal with truth values in numpy arrays...ugh.

I don't want to unintentionally force us to support weird edge case behavior with groupby. On the upside, this got me to write a nice little test utility to neatly check that something raises a warning.

jtratner · 2013-06-08T20:26:32Z

Okay, so this now has test cases & such. I'd vote to remove jtratner@d1ce722 (which specifically tests the results), but I wanted to include it if you want it.

I also added a new test utility to check for warnings (jtratner@86d56ac ). Given that it is a wrapper around a context manager and that it explicitly changes warnings.simplefilter, I felt it made more sense to use as a context manager, especially because it means that you can add additional tests around the warning if you want.

Doctests on the test utility all pass as well. (not sure if that' s included with the Travis tests).

hayd · 2013-06-08T20:42:36Z

Yeah, you're right about removing jtratner@d1ce722 (sorry!).

assert_produces_warning looks whizzy!

jtratner · 2013-06-08T21:10:38Z

Hahaha, I just finished fixing the test case from the SO question. Removing that forever and then pushing.

jtratner · 2013-06-08T21:22:53Z

Okay, this should all pass + I added notes to RELEASE.rst for this.

jtratner · 2013-06-09T00:49:36Z

Never mind. I'm wrong, needed to reset the warning filter each time in assert_produces_warnings.

hayd · 2013-06-09T00:50:46Z

Did you ever find any way of getting the previous ValueError('Function does not reduce') ?

jtratner · 2013-06-09T01:55:45Z

@hayd Nope. Not at all - very weird.

jreback · 2013-06-09T16:17:11Z

can u post the example that works now (with the warning ) and that didn't work before
with what it outputs?

hayd · 2013-06-09T16:58:51Z

The example from #3788:

In [5]: df = pd.DataFrame([[1,np.array([10,20,30])],
   ...:                [1,np.array([40,50,60])],
   ...:                [2,np.array([20,30,40])],], columns=['category','arraydata'])

In [6]: g = df.groupby('category')

In [7]: g.agg(sum)
pandas/core/groupby.py:890: PerformanceWarning: Function does not produce aggregated values. Will not be able to optimize and may produce unexpected results.
  return self._aggregate_series_pure_python(obj, func)
Out[7]:
             arraydata
category
1         [50, 70, 90]
2         [20, 30, 40]

As an aside, I just threw in list to see what would happen (this is current behaviour in master):

In [8]: df = pd.DataFrame([[1,[10,20,30]],
               [1,[40,50,60]],
               [2,[20,30,40]],], columns=['category','arraydata'])

In [9]: g = df.groupby('category')

In [10]: g.agg(sum)
Out[10]:
                         arraydata
category
1         [10, 20, 30, 40, 50, 60]
2                     [20, 30, 40]

hayd · 2013-06-09T17:01:31Z

Maybe it should read "Function may not produce aggregated values." lol

jtratner · 2013-06-09T18:08:15Z

I don't think it's worth it to change the list part, because to do it means adding additional isinstance calls in the middle of everything. It's really clear that something is going wrong when you do this, so I don't think it's a huge deal to just leave it like this. @jreback example previously raised an exception.

hayd · 2013-06-09T18:33:24Z

I still claim there's no ambiguity in either of those results, imo they both aggregate.

wesm · 2013-06-10T02:40:17Z

Looks like tests are failing? What to do here

jtratner · 2013-06-10T02:44:47Z

@wesm not sure - occurred after I rebased on to the most recent master. Testing for warnings is really finicky.

jtratner · 2013-06-10T03:02:46Z

Made the test function simpler, hopefully it will just work this time.

hayd · 2013-06-10T03:58:34Z

One test failing, and now needs a rebase (maybe worth squashing first).

jtratner · 2013-06-10T04:04:16Z

Well, I've been trying a bunch of different ways to get it to work and none
of them are passing travis. But yeah, when I pull them back, I'll rebase it
and make it cleaner.

On Sun, Jun 9, 2013 at 11:58 PM, Andy Hayden [email protected]:

One test failing, and now needs a rebase (maybe worth squashing first).

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/3805#issuecomment-19179986
.

…e a PerformanceWarning instead warns that a non-aggregating function will result in degraded performance TST: Catch warnings in test_groupby

jtratner · 2013-06-10T04:36:42Z

@hayd can you run this and see what you get? jtratner@dfb13e0 . I can't reproduce the error that Travis CI gets so it's difficult for me to fix it.

hayd · 2013-06-10T10:33:31Z

I can't reproduce either...

jtratner · 2013-06-10T11:31:42Z

I'm thinking it's a Travis error (if you notice, a few other warnings
bubble up inapprorpiately)... Maybe if I change it to raise an error
instead, it will work.

On Mon, Jun 10, 2013 at 6:33 AM, Andy Hayden [email protected]:

I can't reproduce either...

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/3805#issuecomment-19191223
.

jtratner · 2013-06-11T23:47:42Z

I managed to reproduce this on my computer in 2.6! Now hopefully I can figure out why...

jtratner · 2013-06-12T03:52:22Z

Update part 2: never mind, it happened two times on my system and then never failed again - I'm trying to rerun an old commit that worked and then see whether it fails when rebased onto the newest master.

jreback · 2013-06-12T14:10:31Z

@jtratner how's this coming?

jtratner · 2013-06-12T15:17:50Z

@jreback I tried testing the failing example in travis on master - turns
out that it doesn't fail in all versions as expected (in other words, it
doesn't raise the Exception about not aggregating values). So I'm guessing
that something is going on where it goes down different code paths
depending on the version and/or something different about the environment
on Travis. I set up a build that (I think) will trace all the functions
executing in core/groupby, so I can see whether some patterns falls out...

On Wed, Jun 12, 2013 at 10:10 AM, jreback [email protected] wrote:

@jtratner https://github.com/jtratner how's this coming?

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/3805#issuecomment-19327938
.

jtratner · 2013-06-12T23:51:00Z

@jreback I've been watching the tracing and have learned some things: for example, the ValueError('Function does not reduce') actually gets caught within aggregate and needs to raise a ValueError, so you definitely shouldn't merge this as it stands now. Still can't figure out why there's different behavior in groupby for the motivating example in different environments.

jreback · 2013-06-13T00:18:24Z

ok....yeh lots of type inference operations that groupby needs....lmk

jreback · 2013-06-13T18:14:18Z

are we pushing this to 0.12 for consideratino?

jtratner · 2013-06-13T21:07:31Z

That seems reasonable to me, it's relatively low importance and not worth being a blocker for the release.

hayd · 2013-06-14T01:41:41Z

related http://stackoverflow.com/questions/17097942/when-does-df-groupby-aggmyagg-pass-a-series-to-myagg

jtratner · 2013-10-12T01:30:47Z

If someone else wants to pick this up, go ahead. Going to close - I don't think it's worth the time for an edge case like this.

Move PerformanceWarning to core/common.py

061074f

jtratner added 2 commits June 10, 2013 00:31

ENH: Change groupby to allow funcs to produce lists/ndarrays, but giv…

557720d

…e a PerformanceWarning instead warns that a non-aggregating function will result in degraded performance TST: Catch warnings in test_groupby

DOC: Update RELEASE.rst

bbcafd8

jtratner mentioned this pull request Jul 24, 2013

"Function does not reduce" for multiindex, but works fine for single index. #4293

Closed

jtratner mentioned this pull request Aug 23, 2013

BUG: allow as_index=False for non aggregating groupby methods #4649

Closed

jtratner closed this Oct 12, 2013

Allow aggregate funcs to return arrays in groupby #3805

Allow aggregate funcs to return arrays in groupby #3805

Conversation

jtratner commented Jun 7, 2013

hayd commented Jun 8, 2013

hayd commented Jun 8, 2013

jtratner commented Jun 8, 2013

hayd commented Jun 8, 2013

jtratner commented Jun 8, 2013

hayd commented Jun 8, 2013

jtratner commented Jun 8, 2013

hayd commented Jun 8, 2013

jtratner commented Jun 8, 2013

jreback commented Jun 8, 2013

jtratner commented Jun 8, 2013

hayd commented Jun 8, 2013

jreback commented Jun 8, 2013

jtratner commented Jun 8, 2013

jtratner commented Jun 8, 2013

jreback commented Jun 8, 2013

jtratner commented Jun 8, 2013

jtratner commented Jun 8, 2013

jtratner commented Jun 8, 2013

hayd commented Jun 8, 2013

jtratner commented Jun 8, 2013

hayd commented Jun 8, 2013

jtratner commented Jun 8, 2013

jtratner commented Jun 8, 2013

hayd commented Jun 8, 2013

jtratner commented Jun 8, 2013

jtratner commented Jun 8, 2013

jtratner commented Jun 9, 2013

hayd commented Jun 9, 2013

jtratner commented Jun 9, 2013

jreback commented Jun 9, 2013

hayd commented Jun 9, 2013

hayd commented Jun 9, 2013

jtratner commented Jun 9, 2013

hayd commented Jun 9, 2013

wesm commented Jun 10, 2013

jtratner commented Jun 10, 2013

jtratner commented Jun 10, 2013

hayd commented Jun 10, 2013

jtratner commented Jun 10, 2013

jtratner commented Jun 10, 2013

hayd commented Jun 10, 2013

jtratner commented Jun 10, 2013

jtratner commented Jun 11, 2013

jtratner commented Jun 12, 2013

jreback commented Jun 12, 2013

jtratner commented Jun 12, 2013

jtratner commented Jun 12, 2013

jreback commented Jun 13, 2013

jreback commented Jun 13, 2013

jtratner commented Jun 13, 2013

hayd commented Jun 14, 2013

jtratner commented Oct 12, 2013