~jeredsutton_opstack


#76 Error parsing password from yoyo.ini 3 years ago

Comment by ~jeredsutton_opstack on ~olly/yoyo

I think you are correct, however that will be difficult to do with the way the environment variables are being populated.

The issue seems to stem from the code assuming the 'database' URL is properly urlencoded (a valid assumption). If that is the case, can I turn this into a feature request to support a 'username' and/or 'password' option in the config file with those options passing through a urllib.parse.quote?

#76 Error parsing password from yoyo.ini 3 years ago

Ticket created by ~jeredsutton_opstack on ~olly/yoyo

This appears to be a variant of #74.

For the following database connection string:

postgresql://%(DATABASE_USERNAME)s:%(DATABASE_PASSWORD)s@%(DATABASE_HOST)s/%(DATABASE_NAME)s?%(DATABASE_OPTIONS)s

If I provide a password that includes symbols such as:

+&~#/

I receive the following error:

ValueError: invalid literal for int() with base 10: "pass+&~#/"

Sorry to keep bothering, and thank you for taking a look.

#74 Error when running migrations 3 years ago

Comment by ~jeredsutton_opstack on ~olly/yoyo

The yoyo version is 7.2.1

#74 Error when running migrations 3 years ago

Ticket created by ~jeredsutton_opstack on ~olly/yoyo

I am using yoyo to run migrations on AWS Aurora Postgresql. I am attempting to use IAM authentication to the database, however it looks like there may be an error in the connection string/password parsing. See below:

yoyo apply ./migrations --database "postgresql://myuser@abcdef1234.1234abcdef.us-east-1.rds.amazonaws.com/dbname?port=5432&sslmode=verify-full&sslrootcert=../../../rds-ca-2019-root.pem" -p --no-config-file
Password for postgresql://myuser@abcdef1234.1234abcdef.us-east-1.rds.amazonaws.com/dbname?port=5432&sslmode=verify-full&sslrootcert=../../../rds-ca-2019-root.pem: 
Traceback (most recent call last):
  File "/home/myuser/.local/share/virtualenvs/infrastructure/bin/yoyo", line 8, in <module>
    sys.exit(main())
  File "/home/myuser/.local/share/virtualenvs/infrastructure/lib/python3.7/site-packages/yoyo/scripts/main.py", line 313, in main
    args.func(args, config)
  File "/home/myuser/.local/share/virtualenvs/infrastructure/lib/python3.7/site-packages/yoyo/scripts/migrate.py", line 249, in apply
    backend = get_backend(args, config)
  File "/home/myuser/.local/share/virtualenvs/infrastructure/lib/python3.7/site-packages/yoyo/scripts/main.py", line 275, in get_backend
    return connections.get_backend(dburi, migration_table)
  File "/home/myuser/.local/share/virtualenvs/infrastructure/lib/python3.7/site-packages/yoyo/connections.py", line 88, in get_backend
    parsed = parse_uri(uri)
  File "/home/myuser/.local/share/virtualenvs/infrastructure/lib/python3.7/site-packages/yoyo/connections.py", line 123, in parse_uri
    port=result.port,
  File "/home/myuser/.pyenv/versions/3.7.7/lib/python3.7/urllib/parse.py", line 169, in port
    port = int(port, 10)
ValueError: invalid literal for int() with base 10: 'abcdef1234.1234abcdef.us-east-1.rds.amazonaws.com%3A5432'

IAM authentication for Aurora works by having the user request a temporary password that is used to log into the database the password is long and contains information that Aurora uses to authorize the user. An example is below:

abcdef1234.1234abcdef.us-east-1.rds.amazonaws.com:5432/?Action=connect&DBUser=myser&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<omitted>%2F20210114%2Fus-east-1%2Frds-db%2Faws4_request&X-Amz-Date=20210114T174138Z&X-Amz-Expires=900&X-Amz-SignedHeaders=host&X-Amz-Security-Token=F<omitted>&X-Amz-Signature=f<omitted>

The issue may actually be in psycopg2, but I figured I would start here. Thank you for taking a look.

Steps to replicate:

  1. Create an AWS RDS Postgresql instance
  2. Enable IAM authentication
  3. Retrieve the password for the created user https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.IAMDBAuth.Connecting.AWSCLI.PostgreSQL.html
  4. Attempt to run yoyo migration using the acquired password