~olly/yoyo#95: 
Not possible to connect to SQLite when my app is packaged with pyinstaller

I have a flask application. For simplicity:

from yoyo.backends.core.sqlite3 import SQLiteBackend
from yoyo.backends.core import __all__

app = Flask(__name__, static_folder='static', template_folder='templates')

def __run_migrations():
    db_path = '/op/mydb.db'
    backend = yoyo.get_backend(f"sqlite:///{db_path}")

    migrations_path = resource_path('./migrations/')
    log.info(f"project root: {migrations_path}")
    migrations = yoyo.read_migrations(migrations_path)
    with backend.lock():
        backend.apply_migrations(backend.to_apply(migrations))

if __name__ == '__main__' : 
    __run_migrations()
    app.run()

It works as it should. But when I obfuscate my code with pyobfuscate and package it with pyinstaller, the app does not start. The error:

Traceback (most recent call last):
  File "yoyo\connections.py", line 74, in get_backend
  File "yoyo\backends\base.py", line 570, in get_backend_class
  File "importlib_metadata\__init__.py", line 294, in __getitem__
KeyError: 'sqlite'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<dist/obf/app.py>", line 3, in <module>
  File "<frozen app>", line 318, in <module>
  File "<frozen app>", line 302, in __run_migrations
  File "yoyo\connections.py", line 76, in get_backend
yoyo.connections.BadConnectionURI: Unrecognised database connection scheme 'sqlite'
[18036] Failed to execute script 'app' due to unhandled exception!

This is my app.spec:

# -*- mode: python ; coding: utf-8 -*-

# Add the pytransform folder to the datas list
datas = [
    ("../dist/obf/pytransform", "pytransform"),
    ("../migrations/", "migrations"),
    ("../templates", "templates"),
    ("../static", "static")
]

block_cipher = None

a = Analysis(
    ['../app.py'],
    pathex=['.'],
    binaries=[],
    datas=datas,
    hiddenimports=['yoyo'],
    hookspath=[],
    hooksconfig={},
    runtime_hooks=[],
    excludes=[],
    win_no_prefer_redirects=False,
    win_private_assemblies=False,
    cipher=block_cipher,
    noarchive=False,
)
pyz = PYZ(a.pure, a.zipped_data, cipher=block_cipher)

# for obfuscated code dependencies to be picked up
a.scripts[-1] = 'app', 'dist/obf/app.py', 'PYSOURCE'
for i in range(len(a.pure)):
    if a.pure[i][1].startswith(a.pathex[0]):
        x = a.pure[i][1].replace(a.pathex[0], os.path.abspath('dist/obf'))
        if os.path.exists(x):
            if hasattr(a.pure, '_code_cache'):
                with open(x) as f:
                    a.pure._code_cache[a.pure[i][0]] = compile(f.read(), a.pure[i][1], 'exec')
            a.pure[i] = a.pure[i][0], x, a.pure[i][2]

exe = EXE(
    pyz,
    a.scripts,
    a.binaries,
    a.zipfiles,
    a.datas,
    [],
    name='app',
    debug=False,
    bootloader_ignore_signals=False,
    strip=False,
    upx=True,
    upx_exclude=[],
    runtime_tmpdir=None,
    console=True,
    disable_windowed_traceback=False,
    argv_emulation=False,
    target_arch=None,
    codesign_identity=None,
    entitlements_file=None,
)

coll = COLLECT(
    exe,
    a.binaries,
    a.zipfiles,
    a.datas,
    strip=False,
    upx=True,
    upx_exclude=[],
    name='app'
)

And this is how I obfuscate the code and package it:

pip install -r requirements.txt

pyarmor-7 obfuscate ^
	--recursive ^
	--output=dist\obf ^
	--exclude="__pycache__" ^
	--exclude="*.pyc" ^
	--exclude="venv" ^
	--exclude="config.yaml" ^
	--exclude="requirements.txt" ^
	--exclude="dist" ^
	--exclude="logs" ^
	--exclude="static" ^
	--exclude="templates" ^
	app.py

pyinstaller --clean -y infra\app.spec

Debuggin the issue, I see that when I run the packaged application, the 'yoyo.backend' entry points are not available:

def get_backend_class(name):
    backend_eps = entry_points(group="yoyo.backends")
    return backend_eps[name].load()

Is this an issue of yoyo-migrations not having a hook for pyinstaller to load all the entry points? Or something else? Can someone help please?

Thanks

Status
RESOLVED CLOSED
Submitter
~rugobal
Assigned to
No-one
Submitted
1 year, 4 months ago
Updated
8 months ago
Labels
No labels applied.

~olly REPORTED CLOSED 8 months ago

There don't seem to be any problems with yoyo finding its entry points when running in a regular python environment. I think it's likely to be down to something that pyinstaller or pyarmor is doing, so I'm closing the issue. But if you find a bug in yoyo that causes this then please do open a new issue for it.

Register here or Log in to comment, or comment via email.