BUG: Incorrect result for Timestamp.ceil during clock change

[torfsen]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
pd.Timestamp("2022-10-30 02:30 +02:00", tz="Europe/Berlin").ceil("1H").tz_convert("UTC")
# returns Timestamp('2022-10-30 02:00:00+0000', tz='UTC')

Issue Description

The Europe/Berlin timezone has a clock change on 2022-10-30, with the clock moving backwards to 02:00 when reaching 03:00 for the first time (in local time). Compared to UTC, the clock goes:

UTC	Europe/Berlin
2022-10-29 23:59	2022-10-30 01:59
2022-10-30 00:00	2022-10-30 02:00
2022-10-30 00:59	2022-10-30 02:59
2022-10-30 01:00	2022-10-30 02:00
2022-10-30 01:59	2022-10-30 02:59
2022-10-30 02:00	2022-10-30 03:00

2022-10-30 02:30 +02 (i.e. 2022-10-30 00:30 +00) is during the "first go", i.e. before the clock jumps back (the same local time in the "second go" is 2022-10-30 02:30 +01, i.e. 2022-10-30 01:30 +00). The next full hour is the time when the clock jumps back, hence I expect

pd.Timestamp("2022-10-30 02:30 +02:00", tz="Europe/Berlin").ceil("1H").tz_convert("UTC")

to return 2022-10-30 01:00 +00. Instead, it returns 2022-10-30 02:00 +00.

This breaks the intuitive assumption that the time delta between the input timestamp and the ceiled timestamp is less than the rounding frequency/delta:

pd.Timestamp("2022-10-30 02:30 +02:00", tz="Europe/Berlin").ceil("1H") - pd.Timestamp("2022-10-30 02:30 +02:00", tz="Europe/Berlin")
# returns  Timedelta('0 days 01:30:00')

This might be related to this comment in #23521 in the sense that Pandas seems to ignore that the timezone is Europe/Berlin and hence carries information about clock changes (as opposed to, say, pytz.FixedOffset(120)).

Expected Behavior

I would have expected ceil to return the next logical hour, upholding the assumption that the time delta between the input timestamp and the ceiled timestamp is less than the rounding frequency/delta. At the very least, I would have expected an exception instead of silently returning an incorrect value.

Installed Versions

INSTALLED VERSIONS

commit : 91111fd
python : 3.8.9.final.0
python-bits : 64
OS : Linux
OS-release : 5.4.0-128-generic
Version : #144-Ubuntu SMP Tue Sep 20 11:00:04 UTC 2022
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.5.1
numpy : 1.23.4
pytz : 2022.5
dateutil : 2.8.2
setuptools : 63.1.0
pip : 22.2.2
Cython : None
pytest : 7.1.2
hypothesis : None
sphinx : 4.5.0
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : 2.9.3
jinja2 : 3.1.2
IPython : 8.3.0
pandas_datareader: None
bs4 : 4.11.1
bottleneck : None
brotli : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : None
numba : None
numexpr : 2.8.1
odfpy : None
openpyxl : 3.0.9
pandas_gbq : None
pyarrow : 7.0.0
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : None
snappy : None
sqlalchemy : None
tables : 3.7.0
tabulate : None
xarray : None
xlrd : None
xlwt : None
zstandard : None
tzdata : None

[torfsen]

Note that converting to UTC, calling ceil, and then converting back to Europe/Berlin is a possible workaround:

pd.Timestamp("2022-10-30 02:30 +02:00", tz="Europe/Berlin").ceil("1H")
# Returns the incorrect Timestamp('2022-10-30 03:00:00+0100', tz='Europe/Berlin')

pd.Timestamp("2022-10-30 02:30 +02:00", tz="Europe/Berlin").tz_convert("UTC").ceil("1H").tz_convert("Europe/Berlin")
# Returns the correct Timestamp('2022-10-30 02:00:00+0100', tz='Europe/Berlin')

[torfsen]

Quick addition to the workaround I posted: it seems to be working for the problematic clock change time span, but note that it produces different results than the plain ceil in other cases:

pd.Timestamp('2000-01-03 23:45:00+0100', tz='Europe/Berlin').ceil("1d")
# Returns Timestamp('2000-01-04 00:00:00+0100', tz='Europe/Berlin')
pd.Timestamp('2000-01-03 23:45:00+0100', tz='Europe/Berlin').astimezone("UTC").ceil("1d").astimezone("Europe/Berlin")
# Returns Timestamp('2000-01-04 01:00:00+0100', tz='Europe/Berlin')

[torfsen]

Here is an improved workaround. Given the subtleties around timezones I wouldn't be suprised if it also gets some cases wrong, but at least the ones I checked were handled correctly. As a bonus, it also avoids the issue from #23521, which I otherwise still run into in 1.5.3.

def ceil(t, freq):
    if t.tzinfo is None:
        # There is no issue with tz-naive timestamps
        return t.ceil(freq)

    # Pretend the timestamp is actually in UTC. Note that this is a hard
    # replacement of the timezone, not a timezone conversion! This is
    # important because a normal timezone conversion can change not just
    # the time but also the date, which can screw up the result if `freq`
    # is large enough (e.g. `1D`).
    fake_utc = t.replace(tzinfo=dt.timezone.utc)

    # In UTC there are no clock changes, so no issues with `ceil`
    ceiled_utc = fake_utc.ceil(freq)
    delta = ceiled_utc - fake_utc

    return t + delta

[smarie]

Nice workaround @torfsen !

It seems that I found a similar issue:

ts = pd.Timestamp('2021-10-31 02:55:00+0200', tz='Europe/Paris')
print(ts.ceil("1h"))                          # yields 2021-10-31 03:00:00+01:00 : incorrect !
print(ts + pd.Timedelta("5min"))   # yields 2021-10-31 02:00:00+01:00 : correct ! 

Here also your workaround solves the problem.

It seems like a perfect "wall-time ceiling/flooring/rounding" workaround. However

• if the period is not a divisor of 1h or 30mins (e.g. period="8min" or "59min"),
• if the desired reference is not the wall time (either it is the dst start of day or the non-dst start of day),
• and/or the dst and non-dst offsets only differ from 30mins instead of 1h (e.g. Lord Howe Island timezone),

then this might not work. Finding a general solution to this problem seems quite hard :D

See https://www.timeanddate.com/time/time-zones-interesting.html for fun