Timestamp("now") should match to_datetime("now")

[jbrockmendel]

See #17697, #18666

pd.to_datetime('now') returns the current UTC time, while pd.Timestamp('now') returns the current local time. pd.to_datetime('today') rounds the current UTC time down to midnight, while pd.Timestamp('today') returns the current local time (not rounded down). These should be fixed to follow a single convention.

>>> pd.Timestamp('now')
Timestamp('2017-12-09 12:28:31.642416')
>>> pd.to_datetime('now')
Timestamp('2017-12-09 20:28:35')
>>> pd.Timestamp('today')
Timestamp('2017-12-09 12:28:49.263407')
>>> pd.to_datetime('today')
Timestamp('2017-12-09 00:00:00')

AFAICT the to_datetime behavior was designed to match the behavior of np.datetime64('now') and np.datetime64('today'). The Timestamp behavior matches the special constructors Timestamp.now(), Timestamp.today(). At this point I think the least disruptive change would be to make Timestamp(...) behave like to_datetime(...).

Thoughts?

[jreback]

this needs to match what datetime doesso Timestamp.now()
is correct

[jbrockmendel]

this needs to match what datetime doesso Timestamp.now() is correct

Timestamp.now() needs to match datetime.now(), but it isn't necessarily obvious that Timestamp('now') needs to match those since there is no datetime('now'). Not that I mind that particular behavior, it just doesn't seem obviously necessary.

[jreback]

i would think this is obvious - now should be now; anything else is completely wrong

it should not match np.datetime which does now in the current time zone which is completely wrong

[jbrockmendel]

AFAICT a discussion with strong opinions occurred on this topic long before I got here.

it should not match np.datetime which does now in the current time zone which is completely wrong

This is confusing for a few reasons.

1. np.datetime64('now') returns a UTC-based value but np.datetime64('today') returns a local-based value*.
2. to_datetime returns a tz-naive object (assuming no tz kwarg is passed), but any TZ-aware strings are converted to their UTC values. i.e. to_datetime is implicitly UTC.
3. The stdlib datetime.today() is equivalent to datetime.now(), while one could reasonably expect it to be equivalent to datetime.now().replace(hour=0, minute=0, second=0, microsecond=0)
   3b) datetime.now(tz) does not correspond to the current time in that timezone, but datetime.now().replace(tzinfo=tz)

* Note: I now think I was wrong in the OP when I wrote that it rounds the UTC time down to midnight; it apparently rounds the local time, i.e. np.datetime64('today') behaves like datetime.today().replace(hour=0, minute=0, second=0, microsecond=0)).

Getting back to the hopefully-actionable topic, a summary of the status quo:

• to_datetime('now') is not all that crazy because it is implicitly UTC and satisfies to_datetime('now').value/1e9 == time.time().
• to_datetime('today') is straight up wrong because of that same implicit-UTCness. The only question is if we want it to continue rounding down to midnight (like np.datetime64) or just behave like "now" (like Timestamp).
• Timestamp.now(), Timestamp('now'), Timestamp.today(), and Timestamp('today') are all implicitly local. We should consider making that explicit. This is a departure from the stdlib, but Timestamp.utcnow() already departs from datetime.utcnow().

[jreback]

we should mimic the standard library as close as possible here. most of numpy functions are wrong in that they use the local timezone for things,

 In [22]: np.datetime64('now')
Out[22]: numpy.datetime64('2017-12-10T15:11:17')

is bonkers and wrong.

Timestamp.now(), Timestamp('now'), Timestamp.today(), and Timestamp('today') are all implicitly local. We should consider making that explicit. This is a departure from the stdlib, but Timestamp.utcnow() already departs from datetime.utcnow().

I think this is fine, these give back a naive stamp, its also quite convenient.

[20] is completely bonkers and should just return Timestamp.now, [21] is fine.

In [20]: pd.to_datetime('now')
Out[20]: Timestamp('2017-12-10 15:09:58.887209')

In [21]: pd.to_datetime('now',utc=True)
Out[21]: Timestamp('2017-12-10 15:10:06.704436+0000', tz='UTC')

[miccoli]

I'm quite confused by this behaviour

>>> pd.to_datetime('now')
Timestamp('2021-10-30 16:29:40.394865')
>>> pd.to_datetime('today')
Timestamp('2021-10-30 18:29:44.680075')

so that

• pd.to_datetime('now') is in UTC and matches np.datetime64('now') ,
• pd.to_datetime('today') is in localtime and does not match np.datetime64('today') any more.

Summing up, if I'm interested in the naive current local timestamp, I can use any one of

pd.Timestamp.now(), pd.Timestamp('now'), pd.Timestamp.today(), pd.Timestamp('today'), pd.to_datetime('today')

while pd.to_datetime('now') is naive UTC. Moreover this behavoiur is not documented.

My 2¢ suggestion:

• clearly state in the docs that today() and now() methods are to be considered aliases in pd.Timestamp
• redefine pd.to_datetime('now') to match current pd.to_datetime('today') ora add a BIG warning in the docs.

[jreback]

@miccoli happy to take a PR to fix up - i don't think changing the docs is actually what we want

[miccoli]

OK, I'll a have a look on this: progress can be seen at master...miccoli:bugfix/GH-18705

[miccoli]

Just a quick update: as of pandas-1.3.4 and numpy-1.21.3

• pd.to_datetime("now") matches np.datetime64("now") (implicit UTC)
• pd.to_datetime("today") matches np.datetime64("today") (implicit local time) but with extra precision.

According to numpy/numpy#10003 the current behaviour is undocumented and bound to be changed to be more consistent with stdlib datetime.

Therefore in the upcoming PR I will drop all references to numpy behaviour and check only against datetime.datetime.now() and datetime.datetime.today()