~mcepl/m2crypto#341: 
ASN1_Time is all fouled up on 32bit architectures

Migrated from: https://gitlab.com/m2crypto/m2crypto/-/issues/341
Created by: Matěj Cepl mcepl@cepl.eu
Created at: 2023-10-24T09:24:25.839Z
Milestone: 0.42

4b3a73bd is all wrong

Status
RESOLVED FIXED
Submitter
~mcepl
Assigned to
No-one
Submitted
a month ago
Updated
12 days ago
Labels
milestone-0.42

~mcepl a month ago

Changed on 2023-10-24T15:20:05.396Z by Matěj Cepl:

changed title from ASN1_Time is all fouled up on {-Window-}s to ASN1_Time is all fouled up on {+32bit architecture+}s

(Last edited at 2023-10-24T15:20:05.401Z.)

~mcepl a month ago

Changed on 2023-10-27T16:18:59.666Z by Matěj Cepl:

mentioned in merge request !306

(Last edited at 2023-10-27T16:18:59.673Z.)

~mcepl a month ago

On 2023-10-27T16:26:35.628Z, Matěj Cepl wrote:

One backtrace:

# TAP results for X509TestCase
not ok 335 test_date_after_2050_working (test_x509.X509TestCase.test_date_after_2050_working)
# Traceback (most recent call last):
#   File "/usr/lib/python3.11/unittest/case.py", line 57, in testPartExecutor
#     yield
#   File "/usr/lib/python3.11/unittest/case.py", line 623, in run
#     self._callTestMethod(testMethod)
#   File "/usr/lib/python3.11/unittest/case.py", line 579, in _callTestMethod
#     if method() is not None:
#        ^^^^^^^^
#   File "/home/user/m2crypto/tests/test_x509.py", line 594, in test_date_after_2050_working
#     self.assertEqual(str(cert.get_not_after()), 'Feb  9 14:57:46 2116 GMT')
#                          ^^^^^^^^^^^^^^^^^^^^
#   File "/home/user/m2crypto/build/lib.linux-i686-cpython-311/M2Crypto/X509.py", line 651, in get_not_after
#     out.set_datetime(ref.get_datetime())
#   File "/home/user/m2crypto/build/lib.linux-i686-cpython-311/M2Crypto/ASN1.py", line 273, in set_datetime
#     date = date.astimezone(local)
#            ^^^^^^^^^^^^^^^^^^^^^^
#   File "/home/user/m2crypto/build/lib.linux-i686-cpython-311/M2Crypto/ASN1.py", line 143, in utcoffset
#     if self._isdst(dt):
#        ^^^^^^^^^^^^^^^
#   File "/home/user/m2crypto/build/lib.linux-i686-cpython-311/M2Crypto/ASN1.py", line 164, in _isdst
#     stamp = time.mktime(tt)
#             ^^^^^^^^^^^^^^^
# OverflowError: mktime argument out of range

~mcepl a month ago

On 2023-10-29T21:41:54.601Z, Matěj Cepl wrote:

That’s actually wrong: !306 just skipped over failing tests, it hasn’t fixed them.

Also: the answer may lie somewhere in the macro SIZEOF_TIME_T defined in pyconfig.h.

if (SIZEOF_TIME_T == 4) then
...

(Last edited at 2023-10-29T21:42:56.224Z.)

~mcepl a month ago

On 2023-10-30T07:29:32.291Z, Matěj Cepl wrote:

Less probably something beautiful happens when we look at this construct https://stackoverflow.com/questions/2816046/simple-typemap-example-in-swig-java, but I would prefer that SIZEOF_TIME_T macro.

~mcepl a month ago

Changed on 2024-01-29T18:21:04.387Z by Matěj Cepl:

mentioned in commit 1da7495e57eaa040e34522840fe04554f3a5a52f

(Last edited at 2024-01-29T18:21:04.402Z.)

~mcepl a month ago*

From the current M2Crypto on openSUSE/Tumbleweed (build log):

[  417s] =================================== FAILURES ===================================
[  417s] __________________ X509TestCase.test_date_after_2050_working ___________________
[  417s] 
[  417s] self = <tests.test_x509.X509TestCase testMethod=test_date_after_2050_working>
[  417s] 
[  417s]     @unittest.skipIf(platform.system() == 'Windows', 'Skip on Windows.')
[  417s]     def test_date_after_2050_working(self):
[  417s]         cert = X509.load_cert('tests/bad_date_cert.crt')
[  417s] >       self.assertEqual(str(cert.get_not_after()), 'Feb  9 14:57:46 2116 GMT')
[  417s] 
[  417s] tests/test_x509.py:594: 
[  417s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[  417s] ../../BUILDROOT/python-M2Crypto-0.40.0-0.i386/usr/lib/python3.10/site-packages/M2Crypto/X509.py:651: in get_not_after
[  417s]     out.set_datetime(ref.get_datetime())
[  417s] ../../BUILDROOT/python-M2Crypto-0.40.0-0.i386/usr/lib/python3.10/site-packages/M2Crypto/ASN1.py:273: in set_datetime
[  417s]     date = date.astimezone(local)
[  417s] ../../BUILDROOT/python-M2Crypto-0.40.0-0.i386/usr/lib/python3.10/site-packages/M2Crypto/ASN1.py:143: in utcoffset
[  417s]     if self._isdst(dt):
[  417s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[  417s] 
[  417s] self = <M2Crypto.ASN1.LocalTimezone object at 0xf5bfd0b8>
[  417s] dt = datetime.datetime(2116, 2, 9, 14, 57, 46, tzinfo=<M2Crypto.ASN1.LocalTimezone object at 0xf5bfd0b8>)
[  417s] 
[  417s]     def _isdst(self, dt):
[  417s]         # type: (datetime.datetime) -> bool
[  417s]         tt = (dt.year, dt.month, dt.day,
[  417s]               dt.hour, dt.minute, dt.second,
[  417s]               dt.weekday(), 0, -1)
[  417s] >       stamp = time.mktime(tt)
[  417s] E       OverflowError: mktime argument out of range
[  417s] 
[  417s] ../../BUILDROOT/python-M2Crypto-0.40.0-0.i386/usr/lib/python3.10/site-packages/M2Crypto/ASN1.py:164: OverflowError
[  417s] __________________ X509TestCase.test_date_reference_counting ___________________
[  417s] 
[  417s] self = <tests.test_x509.X509TestCase testMethod=test_date_reference_counting>
[  417s] 
[  417s]     @unittest.skipIf(platform.system() == 'Windows', 'Skip on Windows.')
[  417s]     def test_date_reference_counting(self):
[  417s]         """x509_get_not_before() and x509_get_not_after() return internal
[  417s]         pointers into X509. As the returned ASN1_TIME objects do not store any
[  417s]         reference to the X509 itself, they become invalid when the last
[  417s]         reference to X509 goes out of scope and the underlying memory is freed.
[  417s]     
[  417s]         https://gitlab.com/m2crypto/m2crypto/-/issues/325
[  417s]         """
[  417s]         cert = X509.load_cert('tests/bad_date_cert.crt')
[  417s]         not_before = cert.get_not_before()
[  417s] >       not_after = cert.get_not_after()
[  417s] 
[  417s] tests/test_x509.py:607: 
[  417s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[  417s] ../../BUILDROOT/python-M2Crypto-0.40.0-0.i386/usr/lib/python3.10/site-packages/M2Crypto/X509.py:651: in get_not_after
[  417s]     out.set_datetime(ref.get_datetime())
[  417s] ../../BUILDROOT/python-M2Crypto-0.40.0-0.i386/usr/lib/python3.10/site-packages/M2Crypto/ASN1.py:273: in set_datetime
[  417s]     date = date.astimezone(local)
[  417s] ../../BUILDROOT/python-M2Crypto-0.40.0-0.i386/usr/lib/python3.10/site-packages/M2Crypto/ASN1.py:143: in utcoffset
[  417s]     if self._isdst(dt):
[  417s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[  417s] 
[  417s] self = <M2Crypto.ASN1.LocalTimezone object at 0xf66ae088>
[  417s] dt = datetime.datetime(2116, 2, 9, 14, 57, 46, tzinfo=<M2Crypto.ASN1.LocalTimezone object at 0xf66ae088>)
[  417s] 
[  417s]     def _isdst(self, dt):
[  417s]         # type: (datetime.datetime) -> bool
[  417s]         tt = (dt.year, dt.month, dt.day,
[  417s]               dt.hour, dt.minute, dt.second,
[  417s]               dt.weekday(), 0, -1)
[  417s] >       stamp = time.mktime(tt)
[  417s] E       OverflowError: mktime argument out of range
[  417s] 
[  417s] ../../BUILDROOT/python-M2Crypto-0.40.0-0.i386/usr/lib/python3.10/site-packages/M2Crypto/ASN1.py:164: OverflowError
[  417s] =========================== short test summary info ============================
[  417s] FAILED tests/test_x509.py::X509TestCase::test_date_after_2050_working - Overf...
[  417s] FAILED tests/test_x509.py::X509TestCase::test_date_reference_counting - Overf...
[  417s] ====== 2 failed, 298 passed, 9 skipped, 1 deselected in 266.90s (0:04:26) ======
[  417s] error: Bad exit status from /var/tmp/rpm-tmp.OdfKHS (%check)
[  417s] 
[  417s] RPM build errors:
[  417s]     Bad exit status from /var/tmp/rpm-tmp.OdfKHS (%check)
[  417s] 
[  417s] tumbleweed-pkg.milic.suse.cz failed "build python-M2Crypto.spec" at Fri Jun 14 21:21:06 UTC 2024.
[  417s] 

Build failed with exit code 1
The buildroot was: /var/tmp/build-root/standard-i586

Cleaning the build root may fix the problem or allow you to start debugging from a well-defined state:
  - add '--clean' option to your 'osc build' command
  - run 'osc wipe [--vm-type=...]' prior running your 'osc build' command again
tumbleweed-pkg~/b/d/python-M2Crypto (1M)$ osc lbl

~mcepl 30 days ago*

The same problem [?] in OpenWrt https://github.com/openwrt/packages/issues/23438.

~mcepl 29 days ago

I see so the whole problem between failing and non-failing test_date_after_2050_working and test_date_reference_counting is in the difference between muslc, which uses 64-bit time_t, and glibc which (sometimes?) uses 32bit time_t on i386.

Adam Williamson (edited) 29 days ago* · edit

Building on Fedora Rawhide (python 3.13) on i686, test_mkcert unexpectedly passes:

======================================================================
UNEXPECTED SUCCESS: test_mkcert (test_x509.X509TestCase.test_mkcert)
----------------------------------------------------------------------
Ran 302 tests in 105.401s
FAILED (skipped=9, expected failures=2, unexpected successes=1)

I don't know why, but it seems like you might want to know. I guess we'll have to patch out the expected-failure decorator.

Adam Williamson (he/him/his) Fedora QA Fedora Chat: @adamwill:fedora.im | Mastodon: @adamw@fosstodon.org https://www.happyassassin.net

~mcepl 28 days ago*

On Fri Jun 21, 2024 at 11:41 PM CEST, ~Adam Williamson wrote:

I don't know why, but it seems like you might want to know. I guess we'll have to patch out the expected-failure decorator.

It is all about Y2038-problem and time_t definition. We have never agreed on one solution to this problem, every hardware chip, every operating system (I know about Windows and Linux, I don’t even have any idea what Mac OS X or *BSD do with it), and even on Linux every library (namely glibc and musl libc, and there is Alpine Linux, so we cannot ignore it) are doing something different, and the result is absolute XXXX (I cannot use any polite word about it). It is horrible to deal with it in C (by using some esoteric macros), but it is even worse in Python, where I cannot use those macros.

We have https://www.gnu.org/software/libc/manual/html_node/Time-Types.html and https://musl.libc.org/time64.html which are very hard to reconile.

Look at https://gitlab.com/m2crypto/m2crypto/-/jobs/7118074934 this is on i386/alpine and this test (using just standard Python API, socket.setsockopts and struct.struct, there is actually not even M2Crypto truly involved! It is all about definition of timeval struct, which is again … what? ... ambiguous) I just cannot make work on it. In desperation I switched back to i386/debian Docker container, and we have back test_date_after_2050_working and test_date_reference_counting failing.

I have tendency in desperation just to skip those tests on 32bit (and then there is test_mkcert, which is suddenly not failing), but that be just an admission that M2Crypto is broken (see that those failing tests are actually a bit important and part of the core mission of M2Crypto) and I don’t want to give up on that yet.

And let me just not mention that we don’t have a fool-proof definition of what actually 32bit is: (struct.calcsize("P") * 8) == 32 and sys.maxsize > 2**32 are I suspect again subtly different on different platforms).

Let me just go to the corner and cry.

(note: this is Markdown, so adjust your expectations accordingly)

Adam Williamson 28 days ago · edit

On Sat, 2024-06-22 at 06:16 +0000, ~mcepl wrote:

On Fri Jun 21, 2024 at 11:41 PM CEST, ~Adam Williamson wrote:

I don't know why, but it seems like you might want to know. I guess we'll have to patch out the expected-failure decorator.

It is all about Y2038-problem and time_t definition. We have never agreed on one solution to this problem, every hardware chip, every operating system (I know about Windows and Linux, I don’t even have any idea what Mac OS X or *BSD do with it), and even on Linux every library (namely glibc and musl libc, and there is Alpine Linux, so we cannot ignore it) are doing something different, and the result is absolute XXXX (I cannot use any polite word about it). It is horrible to deal with it in C (by using some esoteric macros), but it is even worse in Python, where I cannot use those macros.

We have https://www.gnu.org/software/libc/manual/html_node/Time-Types.html and https://musl.libc.org/time64.html which are very hard to reconile.

Look at https://gitlab.com/m2crypto/m2crypto/-/jobs/7118074934 this is on i386/alpine and this test (using just standard Python API, socket.setsockopts and struct.struct, there is actually not even M2Crypto truly involved!) I just cannot make work on it. In desperation I switched back to i386/debian Docker container, and we have back test_date_after_2050_working and test_date_reference_counting failing.

I have tendency in desperation just to skip those tests on 32bit (and then there is test_mkcert, which is suddenly not failing [1]), but that be just an admission that M2Crypto is broken (see that those failing tests are actually a bit important and part of the core mission of M2Crypto) and I don’t want to give up on that yet.

And let me just not mention that we don’t have a fool-proof definition of what actually 32bit is: (struct.calcsize("P") * 8) == 32 and sys.maxsize > 2**32 are I suspect again subtly different on different platforms).

Let me just go to the corner and cry.

[1] https://gitlab.com/m2crypto/m2crypto/-/jobs/7154396846

#I mean, personally I'd just declare you no longer support 32-bit platforms and go for a beer. :)

Adam Williamson (he/him/his) Fedora QA Fedora Chat: @adamwill:fedora.im | Mastodon: @adamw@fosstodon.org https://www.happyassassin.net

~mcepl 28 days ago*

On Sat Jun 22, 2024 at 8:27 AM CEST, ~Adam Williamson wrote:

I mean, personally I'd just declare you no longer support 32-bit platforms and go for a beer. :)

That you can do in Fedora, but the main purpose of M2Crypto as a project is to support legacy applications (https://xkcd.com/2347/ comes to mind), and there are still plenty of for example routers and IoT things, which are 32bit.

And, of course, then there is Windows. I have never found good numbers on the proportion of 32bit or 64bit Windows right now.

~mcepl 17 days ago*

Almost, everything works except for this:

import socket
import struct
from M2Crypto import SSL

ctx = SSL.Context()
s = SSL.Connection(ctx)

timeout = SSL.timeout()
print(f"timeout.sec = {timeout.sec}")
print(f"timeout.microsec = {timeout.microsec}")
print(f"timeout.pack() = {timeout.pack()}")
print(f"socket.SOL_SOCKET = {socket.SOL_SOCKET}")
print(f"socket.SO_RCVTIMEO = {socket.SO_RCVTIMEO}")
s.socket.setsockopt(socket.SOL_SOCKET, socket.SO_RCVTIMEO, timeout.pack())

passes on Linux/glibc, but fails on Alpine/musl/i386.

The only difference I can observe is that socket.SO_RCVTIMEO is 20 with glibc, but 66 with musl.

~mcepl 16 days ago

So, the conclusion from https://discuss.python.org is that this really could be a musl libc bug.

~mcepl REPORTED FIXED 16 days ago

Matěj Cepl referenced this ticket in commit d6f6c78.

~mcepl 12 days ago

Matěj Cepl referenced this ticket in commit 0e01189.

Register here or Log in to comment, or comment via email.