Ftputil doesn't correctly list the content for some sites such as ftp.gnome.org .
>>> import ftputil
>>> conn=ftputil.FTPHost('ftp.gnome.org','anonymous','pippo@pippo.com')
>>> conn.listdir('/')
['pub']
(should be ['HEADER.html', 'Public', 'about', 'cdimage', 'conspiracy', 'debian', 'debian-cd', 'debian-non-US', 'favicon.ico', 'mirror', 'pub' , 'releases', 'robots.txt', 'ubuntu', 'welcome.msg', 'welcome2.msg']).
I think the problem is derived from python standard ftplib
>>> ftp = ftplib.FTP('ftp.gnome.org')
>>> ftp.login()
'230 Login successful.'
>>> ftp.dir('/')
drwxr-xr-x 4 ftp ftp 15 Dec 19 14:31 pub
However with the standard python library there is the following workaround:
>>> ftp.cwd(path)
>>> ftp.dir()
To apply this to ftputil it's enough to modify _robust_ftp_command method (line 562 in ftputil.py) and always use the code for 'special_case'.
If this small modification doesn't break anything I think it could be applied to ftputil,
regards Nicola
Please note that the suggested workaround break the directory listing for ftp host suck as ftp.sunfreeware.com,
regards Nicola
Hi Nicola,
Thanks for reporting! :)
That server doesn't seem to like slashes in the path:
>>> conn.dir("") -rw-r--r-- 1 ftp ftp 1001 May 07 2007 HEADER.html lrwxrwxrwx 1 ftp ftp 3 Sep 18 2008 Public -> pub drwxr-xr-x 2 ftp ftp 8 Apr 15 18:43 about ... >>> conn.dir("/pub") >>> conn.dir("pub") lrwxrwxrwx 1 ftp ftp 28 Sep 19 2008 EFLIB -> ../mirror/ftp.sunet.se/EFLIB lrwxrwxrwx 1 ftp ftp 19 Sep 19 2008 GNOME -> ../mirror/gnome.org lrwxrwxrwx 1 ftp ftp 14 Sep 19 2008 HEADER.html -> ../HEADER.html ... >>> conn.dir("/mirror") >>> conn.dir("mirror") lrwxrwxrwx 1 ftp ftp 14 Sep 18 2008 HEADER.html -> ../HEADER.html drwxr-xr-x 3 ftp ftp 25 Dec 23 2006 bittornado drwxr-xr-x 11 ftp ftp 13 May 09 09:00 cdimage.ubuntu.com ... >>> conn.dir(".") -rw-r--r-- 1 ftp ftp 1001 May 07 2007 HEADER.html lrwxrwxrwx 1 ftp ftp 3 Sep 18 2008 Public -> pub drwxr-xr-x 2 ftp ftp 8 Apr 15 18:43 about ... >>> conn.dir("../../..") drwxr-xr-x 4 ftp ftp 15 Dec 19 14:31 pub >>> conn.dir("../") >>> conn.dir("../.") >>>You said the workaround to change to the directory first didn't work with ftp.sunfreeware.com. What did you try, what was the outcome, and what did you expect? Please show some code in the interpreter.
Stefan
Hi Stefan,
sorry for the late response I see only now your response, here is the code:
standard ftputil special_case is not ever applied
In [1]: import ftputil In [2]: conn=ftputil.FTPHost('ftp.gnome.org','anonymous','pippo@pippo.com') In [3]: conn.listdir('/') Out[3]: ['pub'] In [4]: conn=ftputil.FTPHost('ftp.sunfreeware.com','anonymous','pippo@pippo.com') In [5]: conn.listdir('/') Out[5]: ['bin', 'dev', 'etc', 'pub', 'usr']patched ftputil special_case is always True:
import ftputil In [2]: conn=ftputil.FTPHost('ftp.gnome.org','anonymous','pippo@pippo.com') In [3]: conn.listdir('/') Out[3]: ['HEADER.html', 'Public', 'about', 'cdimage', 'conspiracy', 'debian', 'debian-cd', 'debian-non-US', 'favicon.ico', 'mirror', 'pub', 'releases', 'robots.txt', 'ubuntu', 'welcome.msg', 'welcome2.msg'] In [4]: conn=ftputil.FTPHost('ftp.sunfreeware.com','anonymous','pippo@pippo.com') In [5]: In [6]: In [7]: conn.listdir('/') --------------------------------------------------------------------------- ParserError Traceback (most recent call last) /home/nicola/workspace/FtpManager/<ipython console> /home/nicola/workspace/FtpManager/extlib/ftputil/ftputil.py in listdir(self, path) 806 any of the available parsers raise a `ParserError`. 807 """ --> 808 return self._stat.listdir(path) 809 810 def lstat(self, path, _exception_for_missing_path=True): /home/nicola/workspace/FtpManager/extlib/ftputil/ftp_stat.pyc in listdir(self, path) 563 the server (e. g. timeout). 564 """ --> 565 return self.__call_with_parser_retry(self._real_listdir, path) 566 567 def lstat(self, path, _exception_for_missing_path=True): /home/nicola/workspace/FtpManager/extlib/ftputil/ftp_stat.pyc in __call_with_parser_retry(self, method, *args, **kwargs) 551 self._allow_parser_switching = False 552 self._parser = MSParser() --> 553 return method(*args, **kwargs) 554 else: 555 raise /home/nicola/workspace/FtpManager/extlib/ftputil/ftp_stat.pyc in _real_listdir(self, path) 418 # correct timestamp values in the cache 419 stat_result = self._parser.parse_line(line, --> 420 self._host.time_shift()) 421 loop_path = self._path.join(path, stat_result._st_name) 422 self._lstat_cache[loop_path] = stat_result /home/nicola/workspace/FtpManager/extlib/ftputil/ftp_stat.pyc in parse_line(self, line, time_shift) 349 st_size = int(dir_or_size) 350 except ValueError: --> 351 raise ftp_error.ParserError("invalid size %s" % dir_or_size) 352 else: 353 st_size = None ParserError: invalid size staff Debugging info: ftputil 2.4.1, Python 2.4.3 (linux2)Nicola
Hi Nicola,
thanks for following up. :-)
I tracked the problem down to this:
>>> import ftplib >>> f = ftplib.FTP("ftp.sunfreeware.com", "anonymous", "sschwarzer@sschwarzer.net") >>> f.dir("/") total 10 lrwxrwxrwx 1 staff 7 Aug 13 2003 bin -> usr/bin d--x--x--x 2 staff 512 Sep 24 2000 dev d--x--x--x 3 staff 512 Sep 25 2000 etc dr-xr-xr-x 3 staff 512 Oct 3 2000 pub d--x--x--x 5 staff 512 Oct 3 2000 usr >>> f.dir(".") lrwxrwxrwx 1 staff 7 Aug 13 2003 bin -> usr/bin dev: total 10 etc: total 10 pub: total 4 -rw-r--r-- 1 staff 74 Sep 25 2000 .message ---------- 1 staff 0 Aug 16 2003 .notar drwxr-xr-x 12 ftp 512 Nov 23 2008 freeware usr: total 4 >>>With the absolute path "/" as argument, the listing is retrieved as expected. With the relative path "." the server recurses down the current directory. I'll see if I can find a clean way to deal with this.
Stefan
Seen from a slightly more low-level perspective:
$ ftp -d ftp.sunfreeware.com ... ftp> dir / ftp: setsockopt (ignored): Permission denied ---> PORT 192,168,178,128,195,19 200 PORT command successful. ---> LIST / 150 Opening ASCII mode data connection for /bin/ls. total 10 lrwxrwxrwx 1 staff 7 Aug 13 2003 bin -> usr/bin d--x--x--x 2 staff 512 Sep 24 2000 dev d--x--x--x 3 staff 512 Sep 25 2000 etc dr-xr-xr-x 3 staff 512 Oct 3 2000 pub d--x--x--x 5 staff 512 Oct 3 2000 usr 226 Transfer complete. ftp> dir . ftp: setsockopt (ignored): Permission denied ---> PORT 192,168,178,128,198,136 200 PORT command successful. ---> LIST . 150 Opening ASCII mode data connection for /bin/ls. lrwxrwxrwx 1 staff 7 Aug 13 2003 bin -> usr/bin dev: total 10 etc: total 10 pub: total 4 -rw-r--r-- 1 staff 74 Sep 25 2000 .message ---------- 1 staff 0 Aug 16 2003 .notar drwxr-xr-x 12 ftp 512 Nov 23 2008 freeware usr: total 4 226 Transfer complete. ftp> dir ftp: setsockopt (ignored): Permission denied ---> PORT 192,168,178,128,228,219 200 PORT command successful. ---> LIST 150 Opening ASCII mode data connection for /bin/ls. total 10 lrwxrwxrwx 1 staff 7 Aug 13 2003 bin -> usr/bin d--x--x--x 2 staff 512 Sep 24 2000 dev d--x--x--x 3 staff 512 Sep 25 2000 etc dr-xr-xr-x 3 staff 512 Oct 3 2000 pub d--x--x--x 5 staff 512 Oct 3 2000 usr 226 Transfer complete.Using no arguments after
dirgives the same output as with "/" as argument whereas using "." as argument triggers the recursion.
Hi Nicola,
I committed a changeset 876 which should fix the problem. The former "special case" is now the default as you had originally suggested;
FTPHost._robust_ftp_commandis simplified a bit.Please get the corresponding version of
ftputil.py, test it and give feedback. Note: You only needftputil.py; the other changes are only to test code. Thanks for your help! :o)Stefan
Note: The patch should fix both problems, both ftp.gnome.org and ftp.sunfreeware.com should now behave as expected.
Thanks Stefan,
the fix seems fine,
Nicola