Migrated from: https://gitlab.com/m2crypto/m2crypto/-/issues/341
Created by: Matěj Cepl mcepl@cepl.eu
Created at: 2023-10-24T09:24:25.839Z
Milestone: 0.42
4b3a73bd is all wrong
Changed on 2023-10-24T15:20:05.396Z by Matěj Cepl:
changed title from ASN1_Time is all fouled up on {-Window-}s to ASN1_Time is all fouled up on {+32bit architecture+}s
(Last edited at 2023-10-24T15:20:05.401Z.)
Changed on 2023-10-27T16:18:59.666Z by Matěj Cepl:
mentioned in merge request !306
(Last edited at 2023-10-27T16:18:59.673Z.)
On 2023-10-27T16:26:35.628Z, Matěj Cepl wrote:
One backtrace:
# TAP results for X509TestCase not ok 335 test_date_after_2050_working (test_x509.X509TestCase.test_date_after_2050_working) # Traceback (most recent call last): # File "/usr/lib/python3.11/unittest/case.py", line 57, in testPartExecutor # yield # File "/usr/lib/python3.11/unittest/case.py", line 623, in run # self._callTestMethod(testMethod) # File "/usr/lib/python3.11/unittest/case.py", line 579, in _callTestMethod # if method() is not None: # ^^^^^^^^ # File "/home/user/m2crypto/tests/test_x509.py", line 594, in test_date_after_2050_working # self.assertEqual(str(cert.get_not_after()), 'Feb 9 14:57:46 2116 GMT') # ^^^^^^^^^^^^^^^^^^^^ # File "/home/user/m2crypto/build/lib.linux-i686-cpython-311/M2Crypto/X509.py", line 651, in get_not_after # out.set_datetime(ref.get_datetime()) # File "/home/user/m2crypto/build/lib.linux-i686-cpython-311/M2Crypto/ASN1.py", line 273, in set_datetime # date = date.astimezone(local) # ^^^^^^^^^^^^^^^^^^^^^^ # File "/home/user/m2crypto/build/lib.linux-i686-cpython-311/M2Crypto/ASN1.py", line 143, in utcoffset # if self._isdst(dt): # ^^^^^^^^^^^^^^^ # File "/home/user/m2crypto/build/lib.linux-i686-cpython-311/M2Crypto/ASN1.py", line 164, in _isdst # stamp = time.mktime(tt) # ^^^^^^^^^^^^^^^ # OverflowError: mktime argument out of range
On 2023-10-29T21:41:54.601Z, Matěj Cepl wrote:
That’s actually wrong: !306 just skipped over failing tests, it hasn’t fixed them.
Also: the answer may lie somewhere in the macro
SIZEOF_TIME_Tdefined in pyconfig.h.if (SIZEOF_TIME_T == 4) then ...(Last edited at 2023-10-29T21:42:56.224Z.)
On 2023-10-30T07:29:32.291Z, Matěj Cepl wrote:
Less probably something beautiful happens when we look at this construct https://stackoverflow.com/questions/2816046/simple-typemap-example-in-swig-java, but I would prefer that
SIZEOF_TIME_Tmacro.
Changed on 2024-01-29T18:21:04.387Z by Matěj Cepl:
mentioned in commit 1da7495e57eaa040e34522840fe04554f3a5a52f
(Last edited at 2024-01-29T18:21:04.402Z.)
From the current M2Crypto on openSUSE/Tumbleweed (build log):
[ 417s] =================================== FAILURES =================================== [ 417s] __________________ X509TestCase.test_date_after_2050_working ___________________ [ 417s] [ 417s] self = <tests.test_x509.X509TestCase testMethod=test_date_after_2050_working> [ 417s] [ 417s] @unittest.skipIf(platform.system() == 'Windows', 'Skip on Windows.') [ 417s] def test_date_after_2050_working(self): [ 417s] cert = X509.load_cert('tests/bad_date_cert.crt') [ 417s] > self.assertEqual(str(cert.get_not_after()), 'Feb 9 14:57:46 2116 GMT') [ 417s] [ 417s] tests/test_x509.py:594: [ 417s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [ 417s] ../../BUILDROOT/python-M2Crypto-0.40.0-0.i386/usr/lib/python3.10/site-packages/M2Crypto/X509.py:651: in get_not_after [ 417s] out.set_datetime(ref.get_datetime()) [ 417s] ../../BUILDROOT/python-M2Crypto-0.40.0-0.i386/usr/lib/python3.10/site-packages/M2Crypto/ASN1.py:273: in set_datetime [ 417s] date = date.astimezone(local) [ 417s] ../../BUILDROOT/python-M2Crypto-0.40.0-0.i386/usr/lib/python3.10/site-packages/M2Crypto/ASN1.py:143: in utcoffset [ 417s] if self._isdst(dt): [ 417s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [ 417s] [ 417s] self = <M2Crypto.ASN1.LocalTimezone object at 0xf5bfd0b8> [ 417s] dt = datetime.datetime(2116, 2, 9, 14, 57, 46, tzinfo=<M2Crypto.ASN1.LocalTimezone object at 0xf5bfd0b8>) [ 417s] [ 417s] def _isdst(self, dt): [ 417s] # type: (datetime.datetime) -> bool [ 417s] tt = (dt.year, dt.month, dt.day, [ 417s] dt.hour, dt.minute, dt.second, [ 417s] dt.weekday(), 0, -1) [ 417s] > stamp = time.mktime(tt) [ 417s] E OverflowError: mktime argument out of range [ 417s] [ 417s] ../../BUILDROOT/python-M2Crypto-0.40.0-0.i386/usr/lib/python3.10/site-packages/M2Crypto/ASN1.py:164: OverflowError [ 417s] __________________ X509TestCase.test_date_reference_counting ___________________ [ 417s] [ 417s] self = <tests.test_x509.X509TestCase testMethod=test_date_reference_counting> [ 417s] [ 417s] @unittest.skipIf(platform.system() == 'Windows', 'Skip on Windows.') [ 417s] def test_date_reference_counting(self): [ 417s] """x509_get_not_before() and x509_get_not_after() return internal [ 417s] pointers into X509. As the returned ASN1_TIME objects do not store any [ 417s] reference to the X509 itself, they become invalid when the last [ 417s] reference to X509 goes out of scope and the underlying memory is freed. [ 417s] [ 417s] https://gitlab.com/m2crypto/m2crypto/-/issues/325 [ 417s] """ [ 417s] cert = X509.load_cert('tests/bad_date_cert.crt') [ 417s] not_before = cert.get_not_before() [ 417s] > not_after = cert.get_not_after() [ 417s] [ 417s] tests/test_x509.py:607: [ 417s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [ 417s] ../../BUILDROOT/python-M2Crypto-0.40.0-0.i386/usr/lib/python3.10/site-packages/M2Crypto/X509.py:651: in get_not_after [ 417s] out.set_datetime(ref.get_datetime()) [ 417s] ../../BUILDROOT/python-M2Crypto-0.40.0-0.i386/usr/lib/python3.10/site-packages/M2Crypto/ASN1.py:273: in set_datetime [ 417s] date = date.astimezone(local) [ 417s] ../../BUILDROOT/python-M2Crypto-0.40.0-0.i386/usr/lib/python3.10/site-packages/M2Crypto/ASN1.py:143: in utcoffset [ 417s] if self._isdst(dt): [ 417s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [ 417s] [ 417s] self = <M2Crypto.ASN1.LocalTimezone object at 0xf66ae088> [ 417s] dt = datetime.datetime(2116, 2, 9, 14, 57, 46, tzinfo=<M2Crypto.ASN1.LocalTimezone object at 0xf66ae088>) [ 417s] [ 417s] def _isdst(self, dt): [ 417s] # type: (datetime.datetime) -> bool [ 417s] tt = (dt.year, dt.month, dt.day, [ 417s] dt.hour, dt.minute, dt.second, [ 417s] dt.weekday(), 0, -1) [ 417s] > stamp = time.mktime(tt) [ 417s] E OverflowError: mktime argument out of range [ 417s] [ 417s] ../../BUILDROOT/python-M2Crypto-0.40.0-0.i386/usr/lib/python3.10/site-packages/M2Crypto/ASN1.py:164: OverflowError [ 417s] =========================== short test summary info ============================ [ 417s] FAILED tests/test_x509.py::X509TestCase::test_date_after_2050_working - Overf... [ 417s] FAILED tests/test_x509.py::X509TestCase::test_date_reference_counting - Overf... [ 417s] ====== 2 failed, 298 passed, 9 skipped, 1 deselected in 266.90s (0:04:26) ====== [ 417s] error: Bad exit status from /var/tmp/rpm-tmp.OdfKHS (%check) [ 417s] [ 417s] RPM build errors: [ 417s] Bad exit status from /var/tmp/rpm-tmp.OdfKHS (%check) [ 417s] [ 417s] tumbleweed-pkg.milic.suse.cz failed "build python-M2Crypto.spec" at Fri Jun 14 21:21:06 UTC 2024. [ 417s] Build failed with exit code 1 The buildroot was: /var/tmp/build-root/standard-i586 Cleaning the build root may fix the problem or allow you to start debugging from a well-defined state: - add '--clean' option to your 'osc build' command - run 'osc wipe [--vm-type=...]' prior running your 'osc build' command again tumbleweed-pkg~/b/d/python-M2Crypto (1M)$ osc lbl
The same problem [?] in OpenWrt https://github.com/openwrt/packages/issues/23438.
I see so the whole problem between failing and non-failing
test_date_after_2050_workingandtest_date_reference_countingis in the difference betweenmuslc, which uses 64-bittime_t, andglibcwhich (sometimes?) uses 32bittime_toni386.
Building on Fedora Rawhide (python 3.13) on i686, test_mkcert unexpectedly passes:
====================================================================== UNEXPECTED SUCCESS: test_mkcert (test_x509.X509TestCase.test_mkcert) ---------------------------------------------------------------------- Ran 302 tests in 105.401s FAILED (skipped=9, expected failures=2, unexpected successes=1)I don't know why, but it seems like you might want to know. I guess we'll have to patch out the expected-failure decorator.
Adam Williamson (he/him/his) Fedora QA Fedora Chat: @adamwill:fedora.im | Mastodon: @adamw@fosstodon.org https://www.happyassassin.net
On Fri Jun 21, 2024 at 11:41 PM CEST, ~Adam Williamson wrote:
I don't know why, but it seems like you might want to know. I guess we'll have to patch out the expected-failure decorator.
It is all about Y2038-problem and
time_tdefinition. We have never agreed on one solution to this problem, every hardware chip, every operating system (I know about Windows and Linux, I don’t even have any idea what Mac OS X or *BSD do with it), and even on Linux every library (namely glibc and musl libc, and there is Alpine Linux, so we cannot ignore it) are doing something different, and the result is absolute XXXX (I cannot use any polite word about it). It is horrible to deal with it in C (by using some esoteric macros), but it is even worse in Python, where I cannot use those macros.We have https://www.gnu.org/software/libc/manual/html_node/Time-Types.html and https://musl.libc.org/time64.html which are very hard to reconile.
Look at https://gitlab.com/m2crypto/m2crypto/-/jobs/7118074934 this is on i386/alpine and this test (using just standard Python API, socket.setsockopts and struct.struct, there is actually not even M2Crypto truly involved! It is all about definition of
timeval struct, which is again … what? ... ambiguous) I just cannot make work on it. In desperation I switched back to i386/debian Docker container, and we have backtest_date_after_2050_workingandtest_date_reference_countingfailing.I have tendency in desperation just to skip those tests on 32bit (and then there is test_mkcert, which is suddenly not failing), but that be just an admission that M2Crypto is broken (see that those failing tests are actually a bit important and part of the core mission of M2Crypto) and I don’t want to give up on that yet.
And let me just not mention that we don’t have a fool-proof definition of what actually 32bit is:
(struct.calcsize("P") * 8) == 32andsys.maxsize > 2**32are I suspect again subtly different on different platforms).Let me just go to the corner and cry.
(note: this is Markdown, so adjust your expectations accordingly)
On Sat, 2024-06-22 at 06:16 +0000, ~mcepl wrote:
On Fri Jun 21, 2024 at 11:41 PM CEST, ~Adam Williamson wrote:
I don't know why, but it seems like you might want to know. I guess we'll have to patch out the expected-failure decorator.
It is all about Y2038-problem and
time_tdefinition. We have never agreed on one solution to this problem, every hardware chip, every operating system (I know about Windows and Linux, I don’t even have any idea what Mac OS X or *BSD do with it), and even on Linux every library (namely glibc and musl libc, and there is Alpine Linux, so we cannot ignore it) are doing something different, and the result is absolute XXXX (I cannot use any polite word about it). It is horrible to deal with it in C (by using some esoteric macros), but it is even worse in Python, where I cannot use those macros.We have https://www.gnu.org/software/libc/manual/html_node/Time-Types.html and https://musl.libc.org/time64.html which are very hard to reconile.
Look at https://gitlab.com/m2crypto/m2crypto/-/jobs/7118074934 this is on i386/alpine and this test (using just standard Python API, socket.setsockopts and struct.struct, there is actually not even M2Crypto truly involved!) I just cannot make work on it. In desperation I switched back to i386/debian Docker container, and we have back test_date_after_2050_working and test_date_reference_counting failing.
I have tendency in desperation just to skip those tests on 32bit (and then there is test_mkcert, which is suddenly not failing [1]), but that be just an admission that M2Crypto is broken (see that those failing tests are actually a bit important and part of the core mission of M2Crypto) and I don’t want to give up on that yet.
And let me just not mention that we don’t have a fool-proof definition of what actually 32bit is:
(struct.calcsize("P") * 8) == 32andsys.maxsize > 2**32are I suspect again subtly different on different platforms).Let me just go to the corner and cry.
#I mean, personally I'd just declare you no longer support 32-bit platforms and go for a beer. :)
Adam Williamson (he/him/his) Fedora QA Fedora Chat: @adamwill:fedora.im | Mastodon: @adamw@fosstodon.org https://www.happyassassin.net
On Sat Jun 22, 2024 at 8:27 AM CEST, ~Adam Williamson wrote:
I mean, personally I'd just declare you no longer support 32-bit platforms and go for a beer. :)
That you can do in Fedora, but the main purpose of M2Crypto as a project is to support legacy applications (https://xkcd.com/2347/ comes to mind), and there are still plenty of for example routers and IoT things, which are 32bit.
And, of course, then there is Windows. I have never found good numbers on the proportion of 32bit or 64bit Windows right now.
Almost, everything works except for this:
import socket import struct from M2Crypto import SSL ctx = SSL.Context() s = SSL.Connection(ctx) timeout = SSL.timeout() print(f"timeout.sec = {timeout.sec}") print(f"timeout.microsec = {timeout.microsec}") print(f"timeout.pack() = {timeout.pack()}") print(f"socket.SOL_SOCKET = {socket.SOL_SOCKET}") print(f"socket.SO_RCVTIMEO = {socket.SO_RCVTIMEO}") s.socket.setsockopt(socket.SOL_SOCKET, socket.SO_RCVTIMEO, timeout.pack())passes on Linux/glibc, but fails on Alpine/musl/i386.
The only difference I can observe is that
socket.SO_RCVTIMEOis 20 with glibc, but 66 with musl.
So, the conclusion from https://discuss.python.org is that this really could be a musl
libcbug.
Matěj Cepl referenced this ticket in commit d6f6c78.
Matěj Cepl referenced this ticket in commit 0e01189.
Matěj Cepl referenced this ticket in commit f390722.
Matěj Cepl referenced this ticket in commit 0949b6d.
Matěj Cepl referenced this ticket in commit 53c3b30.
Matěj Cepl referenced this ticket in commit f054091.