Create timezone 'aware' datetime objects rather than 'naive'

[hjmallon]

The current std::chrono::system_clock conversion loses information during it's conversion to and from Python datetime objects. In C++ we have a count of time since epoch, and an assumption that when writing that time out we will use the system local timezone. When converted to the python type we do not attach the timezone information used when creating it, so we cannot convert to UTC without extra information (i.e. what was the system timezone when the timestamp was converted from C++ to Python).

Would it make sense to add this?

I also looked at using the C++ 20 std::chrono::utc_clock but it is not supported in some of the compilers I use yet.

[Skylion007]

@hjmallon Definitely seems useful as an optional include similar to how we handle the STL Filesystem library.

[nitul1991]

@hjmallon Are you working on this idea at the moment?

[hjmallon]

I am not working on it actively

[JamesUnicomb]

I am facing a similar issue: the tm_isdst seems to not be handled properly (on Linux Ubuntu at least). Possible to always assume datetimes are UTC time from timegm rather than mktime?

The result of round trip time during a daylight savings time change in AEDT:

c++ code.
m.def("test_chrono2", [](std::chrono::system_clock::time_point t) { return t; });

python code:
print(dt.year, dt.month, dt.day, dt.hour, dt.minute, dt.second, dt.microsecond)
print(dt, m.test_chrono2(dt))

the output:
2021 10 3 2 18 46 677734
2021-10-03 02:18:46.677734 2021-10-03 03:18:46.677734

EDIT:
These changes in pybind chrono fix the issues for me. I guess the fix is this: Assume all incoming datetimes are UTC timestamps.
pybind/chrono.h

    using type = std::chrono::time_point<std::chrono::system_clock, Duration>;
    bool load(handle src, bool) {
        using namespace std::chrono;

        // Lazy initialise the PyDateTime import
        if (!PyDateTimeAPI) {
            PyDateTime_IMPORT;
        }

        if (!src) {
            return false;
        }

        std::tm cal;
        microseconds msecs;

        if (PyDateTime_Check(src.ptr())) {
            cal.tm_sec = PyDateTime_DATE_GET_SECOND(src.ptr());
            cal.tm_min = PyDateTime_DATE_GET_MINUTE(src.ptr());
            cal.tm_hour = PyDateTime_DATE_GET_HOUR(src.ptr());
            cal.tm_mday = PyDateTime_GET_DAY(src.ptr());
            cal.tm_mon = PyDateTime_GET_MONTH(src.ptr()) - 1;
            cal.tm_year = PyDateTime_GET_YEAR(src.ptr()) - 1900;
            cal.tm_isdst = -1;
            msecs = microseconds(PyDateTime_DATE_GET_MICROSECOND(src.ptr()));
        } else if (PyDate_Check(src.ptr())) {
            cal.tm_sec = 0;
            cal.tm_min = 0;
            cal.tm_hour = 0;
            cal.tm_mday = PyDateTime_GET_DAY(src.ptr());
            cal.tm_mon = PyDateTime_GET_MONTH(src.ptr()) - 1;
            cal.tm_year = PyDateTime_GET_YEAR(src.ptr()) - 1900;
            cal.tm_isdst = -1;
            msecs = microseconds(0);
        } else if (PyTime_Check(src.ptr())) {
            cal.tm_sec = PyDateTime_TIME_GET_SECOND(src.ptr());
            cal.tm_min = PyDateTime_TIME_GET_MINUTE(src.ptr());
            cal.tm_hour = PyDateTime_TIME_GET_HOUR(src.ptr());
            cal.tm_mday = 1;  // This date (day, month, year) = (1, 0, 70)
            cal.tm_mon = 0;   // represents 1-Jan-1970, which is the first
            cal.tm_year = 70; // earliest available date for Python's datetime
            cal.tm_isdst = -1;
            msecs = microseconds(PyDateTime_TIME_GET_MICROSECOND(src.ptr()));
        } else {
            return false;
        }

        value = time_point_cast<Duration>(system_clock::from_time_t(timegm(&cal)) + msecs);
        return true;
    }

and my own code:

	... chrono_datetime_to_ymdhmsm(std::chrono::system_clock::time_point datetime) {
        int year, month, day, hour, minute, second, microsecond;

		using namespace std::chrono;

		using us_t = duration<int, std::micro>;
        auto us = duration_cast<us_t>(datetime.time_since_epoch() % seconds(1));
        if (us.count() < 0)
            us += seconds(1);

		std::time_t tt = system_clock::to_time_t(time_point_cast<system_clock::duration>(datetime - us));

		std::tm *localtime_ptr = std::gmtime(&tt);
		std::tm localtime = *localtime_ptr;
		localtime.tm_isdst = 0;

		year = localtime.tm_year + 1900;
		month = localtime.tm_mon + 1;
		day = localtime.tm_mday;
		hour = localtime.tm_hour;
		minute = localtime.tm_min;
		second = localtime.tm_sec;
		microsecond = us.count();

[thomthom]

I'm also running into this. @JamesUnicomb - how did you resolve this for your project? Did you make a copy of the pybind11 chrono header and add your own changes - using that over the one that currently ships with pybind?