HDFStore complaining about 'timedelta64[ns]'

[vfilimonov]

The HDFStore seemed to not accepting timedelta's objects. The code

import datetime as dt
import pandas as pd
s  = pd.Series(pd.date_range('2012-1-1', periods=3, freq='D'))
td = pd.Series([ dt.timedelta(days=i) for i in range(3) ])
df = pd.DataFrame(dict(A = s, B = td))

print df
print df.dtypes

store = pd.HDFStore('store.h5')
store['df'] = df
store.close()

results in the ValueError: unknown type: 'timedelta64[ns]'.

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-27-b5619cf79ed9> in <module>()
      9 
     10 store = pd.HDFStore('store.h5')
---> 11 store['df'] = df
     12 store.close()

/Users/user/.virtual_envs/system/lib/python2.7/site-packages/pandas/io/pytables.pyc in __setitem__(self, key, value)
    239 
    240     def __setitem__(self, key, value):
--> 241         self.put(key, value)
    242 
    243     def __delitem__(self, key):

/Users/user/.virtual_envs/system/lib/python2.7/site-packages/pandas/io/pytables.pyc in put(self, key, value, table, append, **kwargs)
    534             table
    535         """
--> 536         self._write_to_group(key, value, table=table, append=append, **kwargs)
    537 
    538     def remove(self, key, where=None, start=None, stop=None):

/Users/user/.virtual_envs/system/lib/python2.7/site-packages/pandas/io/pytables.pyc in _write_to_group(self, key, value, index, table, append, complib, **kwargs)
    869             raise ValueError('Compression not supported on non-table')
    870 
--> 871         s.write(obj = value, append=append, complib=complib, **kwargs)
    872         if s.is_table and index:
    873             s.create_index(columns = index)

/Users/user/.virtual_envs/system/lib/python2.7/site-packages/pandas/io/pytables.pyc in write(self, obj, **kwargs)
   2003             blk = data.blocks[i]
   2004             # I have no idea why, but writing values before items fixed #2299
-> 2005             self.write_array('block%d_values' % i, blk.values)
   2006             self.write_index('block%d_items' % i, blk.items)
   2007 

/Users/user/.virtual_envs/system/lib/python2.7/site-packages/pandas/io/pytables.pyc in write_array(self, key, value)
   1805                 self.write_array_empty(key, value)
   1806             else:
-> 1807                 self._handle.createArray(self.group, key, value)
   1808 
   1809         getattr(self.group, key)._v_attrs.transposed = transposed

/Users/user/.virtual_envs/system/lib/python2.7/site-packages/tables/file.pyc in createArray(self, where, name, object, title, byteorder, createparents)
    778         parentNode = self._getOrCreatePath(where, createparents)
    779         return Array(parentNode, name,
--> 780                      object=object, title=title, byteorder=byteorder)
    781 
    782 

/Users/user/.virtual_envs/system/lib/python2.7/site-packages/tables/array.pyc in __init__(self, parentNode, name, object, title, byteorder, _log, _atom)
    165         # Ordinary arrays have no filters: leaf is created with default ones.
    166         super(Array, self).__init__(parentNode, name, new, Filters(),
--> 167                                     byteorder, _log)
    168 
    169 

/Users/user/.virtual_envs/system/lib/python2.7/site-packages/tables/leaf.pyc in __init__(self, parentNode, name, new, filters, byteorder, _log)
    261         # is a lazy property that automatically handles their loading.
    262 
--> 263         super(Leaf, self).__init__(parentNode, name, _log)
    264 
    265 

/Users/user/.virtual_envs/system/lib/python2.7/site-packages/tables/node.pyc in __init__(self, parentNode, name, _log)
    248             #   Create or open the node and get its object ID.
    249             if new:
--> 250                 self._v_objectID = self._g_create()
    251             else:
    252                 self._v_objectID = self._g_open()

/Users/user/.virtual_envs/system/lib/python2.7/site-packages/tables/array.pyc in _g_create(self)
    198             # on
    199             (self._v_objectID, self.shape, self.atom) = self._createArray(
--> 200                 nparr, self._v_new_title, self.atom)
    201         except:  #XXX
    202             # Problems creating the Array on disk. Close node and re-raise.

/Users/user/.virtual_envs/system/lib/python2.7/site-packages/tables/hdf5Extension.so in tables.hdf5Extension.Array._createArray (tables/hdf5Extension.c:8214)()

/Users/user/.virtual_envs/system/lib/python2.7/site-packages/tables/atom.pyc in from_dtype(class_, dtype, dflt)
    368             return class_.from_kind('string', itemsize, dtype.shape, dflt)
    369         # Most NumPy types have direct correspondence with PyTables types.
--> 370         return class_.from_type(basedtype.name, dtype.shape, dflt)
    371 
    372     @classmethod

/Users/user/.virtual_envs/system/lib/python2.7/site-packages/tables/atom.pyc in from_type(class_, type, shape, dflt)
    392 
    393         if type not in all_types:
--> 394             raise ValueError("unknown type: %r" % (type,))
    395         kind, itemsize = split_type(type)
    396         return class_.from_kind(kind, itemsize, shape, dflt)

ValueError: unknown type: 'timedelta64[ns]'

OS: OSX 10.8.3
python: 2.7.4
pandas: 0.11.0
numpy: 1.7.1
tables: 2.4.0

UPDATE:
It seemed to be a pytables issue.

[jreback]

not implemented right now
just need some conversion to/from int64 (similar to how datetime64[ns] are done
if u need a work around just store the 2 dates

[aullrich2013]

Can you elaborate on storing the 2 dates? I'm trying to store a timedelta64 as well. Thanks!

[jreback]

2 options till i implement
say td is a column of timedelta64 either of these will work

• td.astype('int64') to serialize, then Series(td_as_int64, dtype='m8[ns]') to deserialize
• store Timestamp('20130101') + td, and then subtract Timestamp('20130101') to deserialize

[aullrich2013]

Great. I've tried in pandas .12 but any nan or NaT types cause issues in the reinterpretation. For the time being I'm using floats which can represent the nan

[jreback]

I wouldn't use floats at all

NaT are serialized just fine as datetime64 s

[jreback]

@vfilimonov pls have a look at #4822

lmk what you think

[vfilimonov]

@jreback unfortunately I don't have that much of python experience and understanding of pandas structure to comment on the code. But querying timedeltas in HDF5 would be extremely useful for me, thanks a lot!

[jreback]

I am going to support queries like:

'A>5d'
'A>5d 00:00:04'
'A>5ms'

anything else ?

[vfilimonov]

I think that this is pretty enough - I cannot imagine other cases. Perhaps "W" for weeks and "Y" for years might be also useful. And months make no sense for timedeltas.

[jreback]

ok...just about to merge this .....can always add conversions, pretty easy, though so far have kept it

to only exact convertibles, e.g. days,hours,min,sec,ms,us,ns....(but you can pass a timedelta object too)
so conversion not a big deal

[vfilimonov]

@jreback many thanks!

[jreback]

closed by #4822