~rwa/geminispace.info#9: 
fix robots.txt parsing of empty lines

robots.txt parsing stops when an empty line is hit

people use empty lines to make their robots.txt more readable

Status
RESOLVED WONT_FIX
Submitter
~rwa
Assigned to
No-one
Submitted
9 months ago
Updated
8 months ago
Labels
No labels applied.

~rwa 9 months ago

upstream issue at pythons robot file parser: https://github.com/python/cpython/issues/57490

~rwa 9 months ago

the issue are actually not empty lines per se, but empty lines in a rule, e.g.

User-agent: *
Disallow: /test

Disallow: /test2

There are discussions if this is valid at all.

Register here or Log in to comment, or comment via email.