all2md.utils.robots_txt

robots.txt validation and caching for respectful web crawling.

This module provides robots.txt checking functionality that follows RFC 9309 (Robot Exclusion Protocol) to ensure respectful access to web resources.

class all2md.utils.robots_txt.RobotsTxtCacheEntry

Bases: object

Cache entry for a parsed robots.txt file.

Parameters:

  • parser (RobotFileParser) – Parsed robots.txt file

  • timestamp (float) – Unix timestamp when the entry was cached

  • fetch_failed (bool) – Whether fetching robots.txt failed (determines allow/disallow behavior)

  • status_code (int | None) – HTTP status code from robots.txt fetch (if applicable)

parser: RobotFileParser
timestamp: float
fetch_failed: bool
status_code: int | None
__init__(parser: RobotFileParser, timestamp: float, fetch_failed: bool, status_code: int | None) → None
class all2md.utils.robots_txt.RobotsTxtChecker

Bases: object

Thread-safe robots.txt checker with TTL-based caching.

This class fetches, parses, and caches robots.txt files to determine whether a given URL can be accessed according to the robot exclusion protocol (RFC 9309).

Parameters:

cache_duration (int) – Cache duration in seconds (default: 3600 = 1 hour)

Notes

  • Uses stdlib urllib.robotparser.RobotFileParser for parsing

  • Thread-safe cache with TTL expiration

  • Respects HTTP status codes:

    • 404: Allow all (no robots.txt)

    • 5xx: Temporarily disallow all

    • Timeout/network error: Allow all (per RFC 9309)

  • Special handling for fetching robots.txt itself (bypasses robots.txt check)

Initialize the robots.txt checker.

__init__(cache_duration: int = 3600) → None

Initialize the robots.txt checker.

can_fetch(url: str, remote_options: RemoteInputOptions, policy: RobotsTxtPolicy) → tuple[bool, float | None]

Check if a URL can be fetched according to robots.txt and policy.

Parameters:

  • url (str) – The URL to check

  • remote_options (RemoteInputOptions) – Remote input options (contains user_agent, timeout, etc.)

  • policy (RobotsTxtPolicy) – Policy for handling robots.txt (“strict”, “warn”, or “ignore”)

Returns:

Tuple of (allowed, crawl_delay) - allowed: Whether the URL can be fetched - crawl_delay: Crawl delay in seconds (None if not specified)

Return type:

tuple[bool, float | None]

Raises:

ValidationError – If policy is “strict” and robots.txt disallows the URL

clear_cache() → None

Clear all cached robots.txt entries.

get_cache_size() → int

Get the current cache size.

Returns:

Number of cached robots.txt entries

Return type:

int

all2md.utils.robots_txt.get_global_checker() → RobotsTxtChecker

Get or create the global robots.txt checker instance.

Returns:

The global singleton instance

Return type:

RobotsTxtChecker