Unwanted conversion from timedelta to float (but not datetime)

[dcolascione]

In the code below, masked assignment of a timedelta series converts the series values to float64. The same code running against a datetime correctly keeps the series dtype.

In [1]: import pandas as pd

In [2]: dseries=pd.Series(pd.date_range('1/1/2011', periods=5, freq='H')); dseries=dseries-dseries[0]; dseries
Out[2]: 
0   00:00:00
1   01:00:00
2   02:00:00
3   03:00:00
4   04:00:00
dtype: timedelta64[ns]

In [3]: dseries[dseries==dseries[0]]=None; dseries
Out[3]: 
0   -9.223372e+18
1    3.600000e+12
2    7.200000e+12
3    1.080000e+13
4    1.440000e+13
dtype: float64

In [4]: dseries=pd.Series(pd.date_range('1/1/2011', periods=5, freq='H')); dseries=dseries; dseries
Out[4]: 
0   2011-01-01 00:00:00
1   2011-01-01 01:00:00
2   2011-01-01 02:00:00
3   2011-01-01 03:00:00
4   2011-01-01 04:00:00
dtype: datetime64[ns]

In [5]: dseries[dseries==dseries[0]]=None; dseries
Out[5]: 
0                   NaT
1   2011-01-01 01:00:00
2   2011-01-01 02:00:00
3   2011-01-01 03:00:00
4   2011-01-01 04:00:00
dtype: datetime64[ns]

[jreback]

show your version of pandas.

[jreback]

this was broken in master as well.

This will patch, can you do a PR with appropriate tests?

(pandas) bash-3.2$ git diff
diff --git a/pandas/core/internals.py b/pandas/core/internals.py
index e537cb2..7a5877d 100644
--- a/pandas/core/internals.py
+++ b/pandas/core/internals.py
@@ -1956,7 +1956,8 @@ class TimeDeltaBlock(DatetimeLikeBlockMixin, IntBlock):
         tipo = maybe_infer_dtype_type(element)
         if tipo is not None:
             return issubclass(tipo.type, np.timedelta64)
-        return isinstance(element, (timedelta, np.timedelta64))
+        return is_integer(element) or isinstance(
+            element, (timedelta, np.timedelta64))
 
     def fillna(self, value, **kwargs):

[jreback]

tests can go
pandas/tests/indexing/test_timedelta.py
add tests setting with each of [None, np.nan, pd.NaT]

[dcolascione]

Thanks for the fast response! I'm not sure I'll be able to get a dev environment up and running soon. There are various versioning conflicts with Cython that make this more than a five minute job. :-(

[jreback]

There are various versioning conflicts with Cython that make this more than a five minute job. :-(

really?

http://pandas.pydata.org/pandas-docs/stable/contributing.html#

(pandas) bash-3.2$ conda create -n foo --file ci/requirements_dev.txt  --dry
Fetching package metadata .............
Solving package specifications: .

Package plan for installation in environment /Users/jreback/miniconda3/envs/foo:

The following NEW packages will be INSTALLED:

    alabaster:                0.7.10-py35hb692fe1_0    defaults   
    asn1crypto:               0.22.0-py35ha010153_1    defaults   
    babel:                    2.5.0-py35h6f05a75_0     defaults   
    boto:                     2.48.0-py35h10f7326_1    defaults   
    boto3:                    1.4.7-py35hd462e4d_0     defaults   
    botocore:                 1.7.20-py35h01c6a2e_0    defaults   
    ca-certificates:          2017.08.26-ha1e5d58_0    defaults   
    certifi:                  2017.11.5-py35hd00889a_0 defaults   
    cffi:                     1.10.0-py35hca2225f_1    defaults   
    chardet:                  3.0.4-py35h16a84c2_1     defaults   
    click:                    6.7-py35hcc65ea6_0       defaults   
    cookies:                  2.2.1-py35_0             defaults   
    cryptography:             2.0.3-py35hde935b2_1     defaults   
....

[zinklandi]

Hey guys!

I think the selection and the comparison which is done like ...

dseries[dseries==dseries[0]]

... produces a unwanted transformation of the series dtype. A good way to transform the series back is the from float or int is the following way:

pd.to_timedelta(dseries[dseries < '30 sec'])

I'm not sure if the transformation is a bug, but I hope that I could help :)

This is also something I found:

I think there's a transformation in the series which transforms a pandas.timedelta into numpy.timedelta..

[jreback]

@SDzinklandi the patch is above if you want to submit a PR