Error while deleting/dropping a series/dataframe entry with timezone aware timestamps. #21761

elboerto · 2018-07-06T09:52:09Z

Code Sample, a copy-pastable example if possible

import pandas as pd
start = pd.Timestamp('20170101', tz='Europe/Berlin')
end = pd.Timestamp('20180101', tz='Europe/Berlin')
index = pd.date_range(start, end, freq='15min')
data = [1 for x in range(len(index))]
series = pd.Series(index=index, data=data)
dataframe = pd.DataFrame(series)
timestamp = pd.Timestamp('201701011515', tz='Europe/Berlin')

# Working
series[timestamp]
dataframe.loc[timestamp]

# Errors
del series[timestamp]
dataframe.drop(timestamp)
series.drop(timestamp)

Problem description

It not possible to delete/drop a series/dataframe entry with timezone aware timestamps, while accessing them is possible.

Expected Output

Deletion of the respective entry.

Output of `pd.show_versions()`

INSTALLED VERSIONS ------------------ commit: None python: 3.6.4.final.0 python-bits: 64 OS: Windows OS-release: 10 machine: AMD64 processor: Intel64 Family 6 Model 142 Stepping 9, GenuineIntel byteorder: little LC_ALL: None LANG: None LOCALE: None.None pandas: 0.23.1 pytest: 3.6.2 pip: 10.0.1 setuptools: 39.2.0 Cython: None numpy: 1.14.5 scipy: 1.1.0 pyarrow: None xarray: None IPython: 6.4.0 sphinx: None patsy: None dateutil: 2.7.3 pytz: 2018.5 blosc: None bottleneck: None tables: None numexpr: None feather: None matplotlib: 2.2.2 openpyxl: None xlrd: None xlwt: None xlsxwriter: None lxml: None bs4: None html5lib: None sqlalchemy: None pymysql: None psycopg2: None jinja2: None s3fs: None fastparquet: None pandas_gbq: None pandas_datareader: None

The text was updated successfully, but these errors were encountered:

gfyoung · 2018-07-06T19:25:06Z

@elboerto : Can you try your code against master ? I can't seem to replicate this there.

gnikonorov · 2018-07-06T23:00:24Z

I was able to reproduce this issue with master's code. I have just installed pandas for development with anaconda as outlined here and received the following output when I ran the code:

(pandas-dev) ✔ ~/projects/pandas-gnikonorov/test [issue_21761|…1]
18:58 $ cat test_21761.py
import pandas as pd
start = pd.Timestamp('20170101', tz='Europe/Berlin')
end = pd.Timestamp('20180101', tz='Europe/Berlin')
index = pd.date_range(start, end, freq='15min')
data = [1 for x in range(len(index))]
series = pd.Series(index=index, data=data)
dataframe = pd.DataFrame(series)
timestamp = pd.Timestamp('201701011515', tz='Europe/Berlin')

# Working
series[timestamp]
dataframe.loc[timestamp]

# Errors
del series[timestamp]
dataframe.drop(timestamp)
series.drop(timestamp)
(pandas-dev) ✔ ~/projects/pandas-gnikonorov/test [issue_21761|…1]
18:58 $ python test_21761.py
Traceback (most recent call last):
  File "test_21761.py", line 15, in <module>
    del series[timestamp]
  File "/home/gnikonorov/anaconda3/envs/pandas-dev/lib/python3.6/site-packages/pandas/core/generic.py", line 2740, in __delitem__
    self._data.delete(key)
  File "/home/gnikonorov/anaconda3/envs/pandas-dev/lib/python3.6/site-packages/pandas/core/internals.py", line 4785, in delete
    self.axes[0] = self.axes[0].delete(loc)
  File "/home/gnikonorov/anaconda3/envs/pandas-dev/lib/python3.6/site-packages/pandas/core/indexes/datetimes.py", line 2225, in delete
    return DatetimeIndex(new_dates, name=self.name, freq=freq, tz=self.tz)
  File "/home/gnikonorov/anaconda3/envs/pandas-dev/lib/python3.6/site-packages/pandas/core/indexes/datetimes.py", line 445, in __new__
    ambiguous=ambiguous)
  File "pandas/_libs/tslibs/conversion.pyx", line 977, in pandas._libs.tslibs.conversion.tz_localize_to_utc
pytz.exceptions.AmbiguousTimeError: Cannot infer dst time from '2017-10-29 02:00:00', try using the 'ambiguous' argument
(pandas-dev) ✘-1 ~/projects/pandas-gnikonorov/test [issue_21761|…1]
18:58 $

I would like to attempt to fix this issue, if possible.

gfyoung · 2018-07-06T23:20:45Z

@gnikonorov : As I said earlier, I am (un)fortunately unable to replicate those errors. What version of pandas are you using i.e. what is your output when you run:

import pandas as pd
pd.__version__

gnikonorov · 2018-07-07T00:24:48Z

@gfyoung, I am running v0.24.0.dev0+243.g30eb48c.

(pandas-dev) ✘-1 ~/projects/pandas-gnikonorov [issue_21761|…2]
20:23 $ python
Python 3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 17:14:51)
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
>>> print(pandas.__version__)
0.24.0.dev0+243.g30eb48c
>>>

gfyoung · 2018-07-07T00:44:07Z

@gnikonorov : Hmmm...that's quite weird.

cc @jreback @jorisvandenbossche @TomAugspurger : Are any of you able to replicate this by any chance?

elboerto · 2018-07-09T08:15:09Z

Version

>>> import pandas
>>> print(pandas.__version__)
0.24.0.dev0+237.g3273309d2

Output:

(pandas-dev) C:\Users\Robert\Documents\Github\pandas-elboerto>python test/test_21761.py
Traceback (most recent call last):
  File "test/test_21761.py", line 15, in <module>
    del series[timestamp]
  File "C:\Users\Robert\AppData\Local\conda\conda\envs\pandas-dev\lib\site-packages\pandas\core\generic.py", line 2740, in __delitem__
    self._data.delete(key)
  File "C:\Users\Robert\AppData\Local\conda\conda\envs\pandas-dev\lib\site-packages\pandas\core\internals.py", line 4785, in delete
    self.axes[0] = self.axes[0].delete(loc)
  File "C:\Users\Robert\AppData\Local\conda\conda\envs\pandas-dev\lib\site-packages\pandas\core\indexes\datetimes.py", line 2225, in delete
    return DatetimeIndex(new_dates, name=self.name, freq=freq, tz=self.tz)
  File "C:\Users\Robert\AppData\Local\conda\conda\envs\pandas-dev\lib\site-packages\pandas\core\indexes\datetimes.py", line 445, in __new__
    ambiguous=ambiguous)
  File "pandas/_libs/tslibs/conversion.pyx", line 977, in pandas._libs.tslibs.conversion.tz_localize_to_utc
    elif is_dst:
pytz.exceptions.AmbiguousTimeError: Cannot infer dst time from '2017-10-29 02:00:00', try using the 'ambiguous' argument
``

TomAugspurger · 2018-07-12T19:20:31Z

I'm also able to reproduce.

…

On Mon, Jul 9, 2018 at 3:15 AM, elboerto ***@***.***> wrote: Version >>> import pandas>>> print(pandas.__version__)0.24.0.dev0+237.g3273309d2 Output: (pandas-dev) C:\Users\Robert\Documents\Github\pandas-elboerto>python test/test_21761.py Traceback (most recent call last): File "test/test_21761.py", line 15, in <module> del series[timestamp] File "C:\Users\Robert\AppData\Local\conda\conda\envs\pandas-dev\lib\site-packages\pandas\core\generic.py", line 2740, in __delitem__ self._data.delete(key) File "C:\Users\Robert\AppData\Local\conda\conda\envs\pandas-dev\lib\site-packages\pandas\core\internals.py", line 4785, in delete self.axes[0] = self.axes[0].delete(loc) File "C:\Users\Robert\AppData\Local\conda\conda\envs\pandas-dev\lib\site-packages\pandas\core\indexes\datetimes.py", line 2225, in delete return DatetimeIndex(new_dates, name=self.name, freq=freq, tz=self.tz) File "C:\Users\Robert\AppData\Local\conda\conda\envs\pandas-dev\lib\site-packages\pandas\core\indexes\datetimes.py", line 445, in __new__ ambiguous=ambiguous) File "pandas/_libs/tslibs/conversion.pyx", line 977, in pandas._libs.tslibs.conversion.tz_localize_to_utc elif is_dst: pytz.exceptions.AmbiguousTimeError: Cannot infer dst time from '2017-10-29 02:00:00', try using the 'ambiguous' argument `` — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#21761 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABQHIuZGsGc7RYVmuV2FvcZzB5yz0goMks5uExEWgaJpZM4VFMTk> .

gfyoung added Testing pandas testing functions or related to the test suite Indexing Related to indexing on series/frames, not to indexes themselves labels Jul 6, 2018

gfyoung added the Datetime Datetime data dtype label Jul 6, 2018

gfyoung added Can't Repro and removed Testing pandas testing functions or related to the test suite labels Jul 7, 2018

mirkoha mentioned this issue Oct 4, 2018

Assigning a value to a series with a tz-aware timestamp during DST switch raises AmbiguousTimeError #22989

Closed

mroeschke mentioned this issue Nov 4, 2018

TST: drop tz-aware timestamp from DatetimIndex with DST transition #23479

Merged

4 tasks

gfyoung added Needs Tests Unit test(s) needed to prevent regressions and removed Can't Repro labels Nov 4, 2018

jreback added this to the 0.24.0 milestone Nov 4, 2018

jreback closed this as completed in #23479 Nov 4, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error while deleting/dropping a series/dataframe entry with timezone aware timestamps. #21761

Error while deleting/dropping a series/dataframe entry with timezone aware timestamps. #21761

elboerto commented Jul 6, 2018 •

edited by gfyoung

Loading

gfyoung commented Jul 6, 2018 •

edited

Loading

gnikonorov commented Jul 6, 2018 •

edited

Loading

gfyoung commented Jul 6, 2018 •

edited

Loading

gnikonorov commented Jul 7, 2018

gfyoung commented Jul 7, 2018

elboerto commented Jul 9, 2018

TomAugspurger commented Jul 12, 2018 via email

Error while deleting/dropping a series/dataframe entry with timezone aware timestamps. #21761

Error while deleting/dropping a series/dataframe entry with timezone aware timestamps. #21761

Comments

elboerto commented Jul 6, 2018 • edited by gfyoung Loading

Code Sample, a copy-pastable example if possible

Problem description

Expected Output

Output of pd.show_versions()

gfyoung commented Jul 6, 2018 • edited Loading

gnikonorov commented Jul 6, 2018 • edited Loading

gfyoung commented Jul 6, 2018 • edited Loading

gnikonorov commented Jul 7, 2018

gfyoung commented Jul 7, 2018

elboerto commented Jul 9, 2018

TomAugspurger commented Jul 12, 2018 via email

elboerto commented Jul 6, 2018 •

edited by gfyoung

Loading

Output of `pd.show_versions()`

gfyoung commented Jul 6, 2018 •

edited

Loading

gnikonorov commented Jul 6, 2018 •

edited

Loading

gfyoung commented Jul 6, 2018 •

edited

Loading