Robots.txt parsing fails when one rule line is invalid #111788
Description
Bug report
Bug description:
The robots.txt parsing fails if one line is not parsable from a robots.txt file. I don't think this is valid behavior. Ideally, non-parsable/invalid lines should be skipped. The norobots-rfc says the same too: Implementors should pay particular attention to the robustness in parsing of the /robots.txt file..
File "/usr/local/lib/python3.11/urllib/robotparser.py", line 123, in parse
entry.rulelines.append(RuleLine(line[1], False))
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/urllib/robotparser.py", line 222, in __init__
path = urllib.parse.urlunparse(urllib.parse.urlparse(path))
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/urllib/parse.py", line 395, in urlparse
splitresult = urlsplit(url, scheme, allow_fragments)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/urllib/parse.py", line 500, in urlsplit
_check_bracketed_host(bracketed_host)
File "/usr/local/lib/python3.11/urllib/parse.py", line 446, in _check_bracketed_host
ip = ipaddress.ip_address(hostname) # Throws Value Error if not IPv6 or IPv4
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/ipaddress.py", line 54, in ip_address
raise ValueError(f'{address!r} does not appear to be an IPv4 or IPv6 address')
ValueError: '[routes.productDetail(product.sku, product.slug)' does not appear to be an IPv4 or IPv6 addressI know [routes.productDetail(product.sku, product.slug) is clearly not a valid URL, but I don't think the whole parsing should error out because of this one line.
CPython versions tested on:
3.11
Operating systems tested on:
Linux
Activity
sunmy2019 commentedon Nov 11, 2023
The feasible solution would be to add a new flag for this function, telling whether the user wants to handle or ignore the error.
gyula-lakatos commentedon Nov 13, 2023
I'm not sure about that to be honest. The
crawl-delayandrequest-rateparameters are only processed if they are valid (no error handling there). I think the same logic should apply to urls. Also, lines are usually skipped if they are in a wrong order. I'm unsure why these errors specifically would demand a flag.aisk commentedon Dec 17, 2023
I think this is not caused by the rule being invalid, but
robotparser's bug.I assume that you are parsing a robot.txt with some thing like this:
This is invalid, because '[' is not quoted, but according to the RFC and what current codes do,
robotparseris robust enough and can deal with unquoted pathes.But this case is more special, the
//[foo]/barcan be interpreted as an URL with invalid IPv6 host "[foo]" as well. And current codes passed it tourllib.parse.urlparsein c70a6ae, just to remove the empty query string "?" or empty fragment "#", so an exception occurs.So I think it's better to using a different way to normalize the path.
pythongh-111788: fix a bug that urllib.robotparser will raise exceptiâĶ
pythongh-111788: Fix parsing and normalization of rules and URLs in râĶ
pythongh-111788: Fix parsing and normalization of rules and URLs in râĶ
serhiy-storchaka commentedon Sep 4, 2025
The issue is more complicated. By applying
quote(),unquote(),parse()andunparse()to the rule paths and the URL we lose the difference between the query separator and the percent-encoded?and lose trailing?(see #88375 which report the part of the issue). Also,parse()can fail or return incorrect result when trying to parse a path (this issue). #138502 fixes this.The
robotparsemodule was based on the old document http://www.robotstxt.org/orig.html. Now there is a new standard, RFC 9309 which specifies details. #138502 does not make therobotparsemodule completely conforming RFC 9309, because some details look unclear and self-contradictory.pythongh-111788: Fix parsing and normalization of rules and URLs in râĶ
gh-88375, gh-111788: Fix parsing errors and normalization in robotparâĶ
pythongh-88375, pythongh-111788: Fix parsing errors and normalizationâĶ
pythongh-88375, pythongh-111788: Fix parsing errors and normalizationâĶ
[3.13] gh-88375, gh-111788: Fix parsing errors and normalization in râĶ
pythongh-88375, pythongh-111788: Fix parsing errors and normalizationâĶ
[3.14] gh-88375, gh-111788: Fix parsing errors and normalization in râĶ