~sschwarzer/ftputil#56: 
FTPHost.__del__ and ftp_path._Path._host wreaks havoc with Python's circular garbage collection

Since FTPHost defines a __del__ method and the ftp_path._Path instances have a reference to the FTPHost, Python's circular reference garbage collection can't cleanup. See ​http://docs.python.org/reference/datamodel.html#object.__del__ for details.

This could represent a significant memory leak for anyone using ftputil in a long running app.

Status
RESOLVED FIXED
Submitter
ftputiluser (unverified)
Assigned to
No-one
Submitted
11 years ago
Updated
11 years ago
Labels
bug library

schwa (unverified) 11 years ago · edit

Fixed description to work with Trac wiki formatting.

schwa (unverified) 11 years ago · edit

Thank you for the bug report!

Replying to ftputiluser:

Since FTPHost defines a __del__ method and the ftp_path._Path instances have a reference to the FTPHost, Python's circular reference garbage collection can't cleanup. See ​http://docs.python.org/reference/datamodel.html#object.__del__ for details.

As far as I understand the documentation, uncollectable cycles occur only if several objects in the cycle have a __del__ method and thus the garbage collector can't decide which one to delete first. As _Path doesn't have a __del__ method, there are cycles, but they can be collected.

This could represent a significant memory leak for anyone using ftputil in a long running app.

I guess even most long-running apps generate one or a few FTPHost instances and use them over and over. Of course, if you instantiated many FTPHost objects, non-collectable cycles would lead to a memory leak.

As it happens, when I wrote a unit test for this bug, I found there is an uncollectable cycle regarding files and/or sockets (even though I had considered mutual references between FTPHost and FTPFile objects). I'll look into this.

schwa (unverified) 11 years ago · edit

I managed to fix the problem in changeset [80a2fc118393e9a57b50b8294d4cf51ac3e1a31a](https://git.sr.ht/~sschwarzer/ftputil/commit/80a2fc118393e9a57b50b8294d4cf51ac3e1a31a "Removed FTPHost.__del__ to fix bug #56. I had partial success ...").

It seems there actually was a problem with the references held by _Path and _Stat but I was unable to find out why the garbage collection didn't work. In the end, I removed the __del__ method from FTPHost because I think that's a cleaner solution than trying to break various cycles explicitly.

Register here or Log in to comment, or comment via email.