~mcepl/m2crypto#202: 
Generated Windows binaries are mess

Migrated from: https://gitlab.com/m2crypto/m2crypto/-/issues/202
Created by: Matěj Cepl mcepl@cepl.eu
Created at: 2018-02-23T23:56:41.869Z
Closed at: 2018-02-28T08:30:09.446Z
Milestone: 0.30

This is what we get:

Environment: PYTHON=C:\Python36, PYTHON_VERSION=3.6.x, PYTHON_ARCH=32, OPENSSL_PATH=C:\OpenSSL-Win32, \
    PYWIN32=pywin32-222.win32-py3.6.exe, PYWIN32_RELEASE=b222
Environment: PYTHON=C:\Python36, PYTHON_VERSION=3.6.x, PYTHON_ARCH=32, OPENSSL_PATH=C:\OpenSSL-1-1-Win32, \
    PYWIN32=pywin32-222.win32-py3.6.exe, PYWIN32_RELEASE=b222
Environment: PYTHON=C:\Python27, PYTHON_VERSION=2.7.x, PYTHON_ARCH=32, OPENSSL_PATH=C:\OpenSSL-Win32, \
    PYWIN32=pywin32-222.win32-py2.7.exe, PYWIN32_RELEASE=b222
Environment: PYTHON=C:\Python27, PYTHON_VERSION=2.7.x, PYTHON_ARCH=32, OPENSSL_PATH=C:\OpenSSL-1-1-Win32, \
    PYWIN32=pywin32-222.win32-py2.7.exe, PYWIN32_RELEASE=b222
Environment: PYTHON=C:\Python27-x64, PYTHON_VERSION=2.7.x, PYTHON_ARCH=64, OPENSSL_PATH=C:\OpenSSL-Win64, \
    PYWIN32=pywin32-222.win-amd64-py2.7.exe, PYWIN32_RELEASE=b222
Environment: PYTHON=C:\Python27-x64, PYTHON_VERSION=2.7.x, PYTHON_ARCH=64, OPENSSL_PATH=C:\OpenSSL-1-1-Win64, \
    PYWIN32=pywin32-222.win-amd64-py2.7.exe, PYWIN32_RELEASE=b222
Environment: PYTHON=C:\Python26, PYTHON_VERSION=2.6.6, PYTHON_ARCH=32, OPENSSL_PATH=C:\OpenSSL-Win32, \
    PYWIN32=pywin32-221.win32-py2.6.exe, PYWIN32_RELEASE=b221

Except OpenSSL-1-1 is the same as OpenSSL (which is 1.1 these days, understandably), we don't have 3.6 for Win64 (I would probably excuse 2.6 for Win64), we have too many options (e.g., lxml has only .whl files, OpenSSL and python only .exe files, and I would prfer .msi files only, because they seem like closest to the packages for Windows). OTOH, do we need to do OpenSSL-1.0.1 at all?

Also, it would be nice, if the individual files were distinguishable by name, so we could have all files in one directory (e.g., introduce version of OpenSSL to the filename).

@dwoz Any more thoughts on this, before I start edit appveyor.yml, please?

Status
RESOLVED FIXED
Submitter
~mcepl
Assigned to
No-one
Submitted
8 months ago
Updated
8 months ago
Labels
No labels applied.

~mcepl 8 months ago

Changed on 2018-02-23T23:57:26.190Z by Matěj Cepl:

changed the description

~mcepl 8 months ago

Changed on 2018-02-23T23:58:30.768Z by Matěj Cepl:

changed the description

~mcepl 8 months ago

On 2018-02-24T01:55:11.998Z, Daniel Wozniak wrote:

I'd like to see some kind of whl make it to PyPi if possible. If it's only going to be one or the other between MSI and EXE. I prefer MSI, some folks might still like EXE too so I have no issues with all three.

About naming, not totally sure, back to my PyPi point thoughts. I wonder if there would be a way to have both 1.1.x and 1.0.x versions coexists as binaries in PyPi. I don't know the answer but it would be cool.

~mcepl 8 months ago

On 2018-02-24T01:56:56.431Z, Daniel Wozniak wrote:

btw, the max allowed artifacts thing was a bug in Appveyor. Their support resolved it for me.

~mcepl 8 months ago

Changed on 2018-02-24T08:17:42.416Z by Matěj Cepl:

created branch 202-generated-win-binaries

~mcepl 8 months ago

Changed on 2018-02-24T08:17:43.009Z by Matěj Cepl:

mentioned in merge request !182

~mcepl 8 months ago

Changed on 2018-02-24T10:22:03.796Z by Matěj Cepl:

mentioned in merge request !183

~mcepl 8 months ago

On 2018-02-24T10:25:53.681Z, Matěj Cepl wrote:

Oh sorry, mea culpa, OpenSSL preinstalled on AppVeyor actually IS OpenSSL 1.0.2l. OK, so that's one problem down.

~mcepl 8 months ago

On 2018-02-26T17:59:14.421Z, Shane Lee wrote:

I would like to see .whl files for the following:

win32-py27
win32-py35
win32-py36
win64-py27
win64-py35
win64-py36

(Last edited at 2018-02-26T17:59:28.427Z.)

~mcepl 8 months ago

On 2018-02-27T06:02:21.614Z, Daniel Wozniak wrote:

Okay, thinking about this more. We're talking about making binary packages and having those binary packages tied to the version of openssl of our choosing (1.1.0g at this time). The whole point of providing these binary packages is to make super easy for users to install so let's just close the gap and include the openssl dlls in our binary packages (for windows). That way users can just install M2Crypto just like any other python module. Users won't need to follow any extra directions to set it up. Any power user that wants to link against their own openssl is free to do so via the source packages. I haven't been able to come up with a downside to this approach so I'm going to push a patch for it, in case everyone is on board.

(Last edited at 2018-02-27T06:03:16.550Z.)

~mcepl 8 months ago

Changed on 2018-02-27T10:20:10.709Z by Daniel Wozniak:

mentioned in merge request !187

~mcepl 8 months ago

On 2018-02-27T10:24:58.002Z, Daniel Wozniak wrote:

@Twangboy & @mcepl See the builds on my !187 merge request for the work from my last comment...

!187 ~ Homicide on them packages!

~mcepl 8 months ago

On 2018-02-27T10:40:05.364Z, Daniel Wozniak wrote:

This should resolve the chocolaty swig install flakiness we've been seeing.

https://github.com/chrisdembia/chocolatey-swig/pull/3

~mcepl 8 months ago

On 2018-02-27T11:11:36.582Z, Matěj Cepl wrote:

The whole point of providing these binary packages is to make super easy for users to install so let's just close the gap and include the openssl dlls in our binary packages (for windows).

I don't like bundling-in OpenSSL to M2Crypto, because that effectively makes me responsible for making timely upgrades of M2Crypto whenever there is a new security bug of OpenSSL. I am not bashing OpenSSL developers, some of them are my colleagues, and I admire them hugely, but still, there is no way how to avoid the fact, there are plenty of security vulnerabilities in OpenSSL. I really don’t want to get on that treadmill.

My thinking lately was exactly in other direction. I am old-fart Linux guy, and I have some experience with packaging stuff, and I have some rather dreadful experience with bundling in libraries (I was around Debian many years ago, when we were going through couple of thousand packages and ripping out bundled-in libz with a security issue; experience I will never forget). I understand, that Windows universe packaging technology is sub-optimal and there is no proper packaging culture, but I would rather try to get most of what is there.

So, why not to make just a choco package of M2Crypto (I believe, its metadata could express dependency on OpenSSL, so for our users it would be still just choco install M2Crypto, right?), and perhaps binary wheels for PyPI, and that’s it? (update: perhaps even binary wheels are a bad idea, because we cannot express dependency on OpenSSL in the way which would make pip install M2Crypto work flawlessly, right?)

If anybody wants anything else (e.g., bundled-in OpenSSL, or .exe installer), then the source code is available, and he can make his own bed and lie in it.

@dwoz, @Twangboy what do you think?

(Last edited at 2018-02-27T11:16:29.852Z.)

~mcepl 8 months ago

On 2018-02-27T17:11:53.153Z, Daniel Wozniak wrote:

@mcepl Your points are valid. I've spent time packaging for Debian as well and agree that nothing beats a proper package management system. Something like a Chocoalty or Nuget package would be nice. My only grip with something like that is it doesn't work through PyPi too. Taking everything into account, I agree if we don't want to bundle the required DLLs; not providing binaries at all is best. Lets just go with a source distribution for PyPi. I can hide my changes behind a flag (--bundle-dlls) so it will at least be fairly simple to make a binary that works if someone wishes.

(Last edited at 2018-02-27T17:13:55.791Z.)

~mcepl 8 months ago

On 2018-02-27T17:29:51.936Z, Matěj Cepl wrote:

Well, I hoped to make you to create and maintain a Chocolatey package ;). Do you know anything about it?

~mcepl 8 months ago

On 2018-02-27T17:43:22.373Z, Daniel Wozniak wrote:

I pushed the '--bundledlls' flag patch. I do not have use for a Chocolatey package at this time. At this point it should be easy enough for anyone who wants to pursue it.

~mcepl 8 months ago

On 2018-02-27T22:55:24.733Z, Mart Sõmermaa wrote:

First, many, many thanks for working on this!

Regarding the possible decision to abandon Windows packages, I would like to point out that Stack Overflow is full of people clamoring for them:

So I for one vote for official packages with bundled OpenSSL - even though the objections are absolutely valid, it's a pragmatic decision and the downsides can be clearly documented.

~mcepl 8 months ago

On 2018-02-27T23:12:32.293Z, Matěj Cepl wrote:

@mrts welcome on board. You know how it is with FLOSS. Never say “somebody should do this” only “I will do it”. Will you?

If you are willing to maintain packages for choco (I guess, it looks from my distance like the most elegant solution), or build those packages with bundled OpenSSL, whatever you want, I am willing to help with advice or changes in code (if necessary).

Whatever you guys on Windows agree, I am for it. However, I won't do the work, I don’t even have Windows computer to work on.

(Last edited at 2018-02-27T23:14:11.359Z.)

~mcepl 8 months ago

On 2018-02-27T23:27:33.044Z, Matěj Cepl wrote:

I guess !187 can go in. Anybody any objections?

~mcepl 8 months ago

On 2018-02-27T23:29:17.303Z, Mart Sõmermaa wrote:

Yay, thanks a lot :)! You make a lot of Windows users happy :).

~mcepl 8 months ago

On 2018-02-27T23:30:33.250Z, Daniel Wozniak wrote:

@mcepl As it stands right now, the ddl bundling is turned off. How about I turn it back on and we merge this. I will maintain these binaries for security patches. I'm confident this won't be a problem.

Does this work?

~mcepl 8 months ago

On 2018-02-28T00:00:24.512Z, Daniel Wozniak wrote:

I've pushed an appveyor.yml with the --builddlls flag enabled. The builds are green.

~mcepl 8 months ago

On 2018-02-28T06:35:41.896Z, Matěj Cepl wrote:

@dwoz As I said. Whatever works for y’all. Thank you very much.

Just to make sure, --builddls flag defaults to false, right?

Let me take one more look through your changes.

(Last edited at 2018-02-28T06:39:00.090Z.)

~mcepl 8 months ago

On 2018-02-28T07:06:29.062Z, Daniel Wozniak wrote:

Yes, the dlls will only get included when that flag --bundledlls is passed to the build_ext or build step. It defaults to false and is only valid on windows platforms.

(Last edited at 2018-02-28T07:07:14.857Z.)

~mcepl 8 months ago

Changed on 2018-02-28T08:30:09.469Z by Matěj Cepl:

closed via commit 81e7de136fde6ec1a88ac0d6cbab62536d4b1107

~mcepl 8 months ago

Changed on 2018-02-28T08:30:09.525Z by Matěj Cepl:

closed via merge request !183

~mcepl 8 months ago

On 2018-02-28T09:07:19.522Z, Mart Sõmermaa wrote:

Thank you both once again :)!

~mcepl 8 months ago

On 2018-02-28T09:08:19.199Z, Matěj Cepl wrote:

I will wait with the release of 0.30 until I somehow resolve #203 so hold your breath just lightly.

~mcepl 8 months ago

Changed on 2018-02-28T09:11:32.187Z by Matěj Cepl:

changed milestone to 0.30

~mcepl referenced this from #203 8 months ago

~mcepl REPORTED FIXED 8 months ago

Register here or Log in to comment, or comment via email.