PERF: categorical rank

[jreback]

xref #15422 (comment)

easy enough after #15422 to rank the categories themselves rather than using expanded values; prob most relevant for object dtypes.

In [15]: s = Series(tm.makeCategoricalIndex(100000))

In [16]: res = Series(np.array(s.cat.rename_categories(Series(s.cat.categories).rank()))).rank()

In [17]: res2 = s.rank()

In [18]: res.equals(res2)
Out[18]: True

In [19]: %timeit Series(np.array(s.cat.rename_categories(Series(s.cat.categories).rank()))).rank()
100 loops, best of 3: 4.39 ms per loop

In [20]: %timeit s.rank()
10 loops, best of 3: 132 ms per loop

[jeetjitsu]

@jreback @jorisvandenbossche : in the _values_for_rank method in Categorical, re-organizing and moving the typecast to float outside the if condition, like below, has the advantage that i can set a single rank function, for categoricals, in the _get_data_algo function in pandas/core/algorithms.py. Which imo is cleaner. Should i move it out or do you think otherwise?

    def _values_for_rank(self):
        from pandas import Series
        if self.ordered:
            values = self.codes
            mask = values == -1
            values = values.astype('float64')
            if mask.any():
                values[mask] = np.nan
        else:
            values = np.array(
                self.rename_categories(Series(self.categories).rank())
            )
        return values


    def _get_data_algo(values, func_map):

        f = None

        if is_float_dtype(values):
            f = func_map['float64']
            values = _ensure_float64(values)
        ...
        elif is_categorical_dtype(values):
            f = func_map['float64']
            values = values._values_for_rank()
        ...

[jreback]

yes going to need some reorg
mainly have to pass in the actual rank args themselves (easy enough just pass them thur as kwargs)

[jorisvandenbossche]

@ikilledthecat You can open a PR with the above change, that will be the easiest to discuss (in any case, the above certainly looks reasonable. A reason to keep the astype in the if condition is to avoid a conversion of the data from int to float when not needed, which will give a (small) performance penalty.)

@jreback why is it needed to pass rank args? The above (or similar) seems OK to me without additional args

[jorisvandenbossche]

Another thing we could do for performance is for the unordered categorical to first check whether the categories are sorted before doing the renaming (from a quick test this checking is much less expensive than the actual renaming). Although that may not be worth the complexity.

[jreback]

so when rank is called on the categories it's fine
but needs na--position in order to order any na

though that will be handled when the categories are re expanded so maybe not needed

yeah makes for sense that way