Skip to content

PERF: Utilize mixed dtypes in df.count() with MultiIndexes #9163

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from

Conversation

qwhelan
Copy link
Contributor

@qwhelan qwhelan commented Dec 28, 2014

@jreback Same underlying cause as #9136 but the solution is a bit more involved here.

Basically, we're transposing a potentially mixed-type frame before calling notnull(frame.values); the same result can be obtained by deferring the transpose until after notnull gives us a non-mixed frame.

Here are the vbench results:

-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
frame_count_level_axis1_mixed_dtypes_multi   |  82.2484 | 1489.9830 |   0.0552 |
frame_count_level_axis0_mixed_dtypes_multi   | 101.2537 | 1737.6347 |   0.0583 |
frame_count_level_axis0_multi                |  51.0643 |  51.3713 |   0.9940 |
frame_count_level_axis1_multi                |  98.5887 |  82.6767 |   1.1925 |
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------

Ratio < 1.0 means the target commit is faster then the baseline.
Seed used: 1234

Target [04ec4c6] : PERF: Utilize mixed dtypes in df.count() with MultiIndexes
Base   [def58c9] : Merge pull request #9128 from hsperr/expanduser

@qwhelan qwhelan force-pushed the master branch 2 times, most recently from 2676b04 to 04ec4c6 Compare December 28, 2014 09:09
@jreback jreback added the Performance Memory or execution speed performance label Dec 28, 2014
@jreback jreback added this to the 0.16.0 milestone Dec 28, 2014
if frame._is_mixed_type:
# Since we have mixed types, calling notnull(frame.values) might
# upcast everything to object
mask = notnull(frame).values.view(np.uint8)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you can take out the .view(...) and the comment. This should work w/o it. (the cython will cast a boolean directly)

@jreback
Copy link
Contributor

jreback commented Dec 28, 2014

minor change. looks good otherwise.

@qwhelan
Copy link
Contributor Author

qwhelan commented Dec 29, 2014

@jreback Fixed and Travis is passing.

@jreback
Copy link
Contributor

jreback commented Dec 29, 2014

merged via 9d4f423

thanks!

@jreback jreback closed this Dec 29, 2014
@qwhelan
Copy link
Contributor Author

qwhelan commented Dec 29, 2014

Awesome, thanks.

@jreback
Copy link
Contributor

jreback commented Dec 29, 2014

no thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Performance Memory or execution speed performance
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants