all2md.utils.network_security

Network security utilities for preventing SSRF attacks.

This module provides comprehensive URL validation and secure HTTP client functionality to prevent Server-Side Request Forgery (SSRF) attacks when fetching remote resources.

The security measures include: - DNS resolution validation for all IP addresses - Blocking private, loopback, and special-use IP ranges - Redirect validation and hop limiting - Content-type and size restrictions - Timeout enforcement

Functions

validate_url_security: Comprehensive URL security validation
create_secure_http_client: Create httpx client with security constraints
fetch_image_securely: Secure image fetching with validation

class all2md.utils.network_security.RateLimiter

Bases: object

Rate limiter for network requests using token bucket algorithm.

Implements both requests-per-second rate limiting and concurrent request limiting. Thread-safe implementation using locks and semaphores.

Parameters:

max_requests_per_second (float, default 10.0) – Maximum number of requests per second
max_concurrent (int, default 5) – Maximum number of concurrent requests

Initialize rate limiter.

__init__(max_requests_per_second: float = 10.0, max_concurrent: int = 5): Initialize rate limiter.

classmethod from_options(network_options: Any) → RateLimiter

Build a rate limiter from a NetworkFetchOptions-like object.

Parameters:: network_options (Any) – Object exposing max_requests_per_second and max_concurrent_requests (typically NetworkFetchOptions).
Returns:: Rate limiter configured from the options.
Return type:: RateLimiter

acquire(timeout: float | None = None) → bool

Acquire permission to make a request.

Blocks until rate limit and concurrency constraints are satisfied.

Parameters:: timeout (float | None, default None) – Maximum time to wait in seconds. None means wait indefinitely.
Returns:: True if acquired, False if timeout
Return type:: bool

release() → None: Release a concurrent request slot.

all2md.utils.network_security.validate_url_security(url: str, allowed_hosts: list[str] | None = None, require_https: bool = True) → None

Validate URL for security before making HTTP requests.

Performs comprehensive security validation including DNS resolution, IP address validation, and hostname allowlist checking.

Parameters:

url (str) – URL to validate
allowed_hosts (list[str] | None, default None) – List of allowed hostnames or CIDR blocks. If None, all hosts allowed (subject to IP restrictions)
require_https (bool, default True) – If True, only HTTPS URLs are allowed

Raises:

NetworkSecurityError – If URL fails security validation

all2md.utils.network_security.create_secure_http_client(timeout: float = 10.0, max_redirects: int = 5, allowed_hosts: list[str] | None = None, require_https: bool = True, user_agent: str | None = None) → Any

Create httpx client with security constraints.

This implementation uses httpx event hooks for robust URL validation instead of fragile transport subclassing. The request hook validates every request URL, including each redirect target, before it is followed.

The redirect limit is enforced natively by httpx (max_redirects), which raises httpx.TooManyRedirects when exceeded. It cannot be checked in a response event hook: httpx invokes response hooks before assigning response.history, so the chain length is never visible there.

Parameters:

timeout (float, default 10.0) – Request timeout in seconds
max_redirects (int, default 5) – Maximum number of redirects to follow
allowed_hosts (list[str] | None, default None) – List of allowed hostnames or CIDR blocks
require_https (bool, default True) – If True, only HTTPS URLs are allowed
user_agent (str | None, default None) – Custom User-Agent header for requests

Returns:

Configured HTTP client with security constraints

Return type:

httpx.Client

all2md.utils.network_security.fetch_content_securely(url: str, allowed_hosts: list[str] | None = None, require_https: bool = True, max_size_bytes: int = 20971520, timeout: float = 10.0, expected_content_types: list[str] | None = None, require_head_success: bool = True, rate_limiter: RateLimiter | None = None, user_agent: str | None = None, bypass_robots_txt: bool = False, max_redirects: int = 5) → bytes

Securely fetch content from URL with streaming and comprehensive validation.

Parameters:

url (str) – URL to fetch content from
allowed_hosts (list[str] | None, default None) – List of allowed hostnames or CIDR blocks
require_https (bool, default True) – If True, only HTTPS URLs are allowed
max_size_bytes (int, default 20MB) – Maximum allowed response size in bytes
timeout (float, default 10.0) – Request timeout in seconds
expected_content_types (list[str] | None, default None) – List of allowed content type prefixes (e.g., [“image/”, “text/”])
require_head_success (bool, default True) – Require a successful HEAD request prior to GET.
rate_limiter (RateLimiter | None, default None) – Optional rate limiter to control request rate and concurrency
user_agent (str | None, default None) – Custom User-Agent header for requests
bypass_robots_txt (bool, default False) – If True, bypass robots.txt checking (used internally when fetching robots.txt itself)
max_redirects (int, default 5) – Maximum number of HTTP redirects to follow

Returns:

Content data

Return type:

bytes

Raises:

NetworkSecurityError – If URL fails security validation or fetch constraints

all2md.utils.network_security.fetch_image_securely(url: str, allowed_hosts: list[str] | None = None, require_https: bool = True, max_size_bytes: int = 20971520, timeout: float = 30.0, require_head_success: bool = True, rate_limiter: RateLimiter | None = None, user_agent: str | None = None, max_redirects: int = 5, expected_content_types: list[str] | None = None) → bytes

Securely fetch image data from URL with comprehensive validation.

This is a convenience wrapper around fetch_content_securely for image downloads.

Parameters:

url (str) – URL to fetch image from
allowed_hosts (list[str] | None, default None) – List of allowed hostnames or CIDR blocks
require_https (bool, default True) – If True, only HTTPS URLs are allowed
max_size_bytes (int, default 20MB) – Maximum allowed response size in bytes
timeout (float, default 10.0) – Request timeout in seconds
require_head_success (bool, default True) – Require a successful HEAD request prior to GET
rate_limiter (RateLimiter | None, default None) – Optional rate limiter to control request rate and concurrency
user_agent (str | None, default None) – Optionally set the user agent for http requests.
max_redirects (int, default 5) – Maximum number of HTTP redirects to follow
expected_content_types (list[str] | None, default None) – List of allowed content type prefixes. Defaults to [“image/”].

Returns:

Image data

Return type:

bytes

Raises:

NetworkSecurityError – If URL fails security validation or fetch constraints

all2md.utils.network_security.fetch_image_with_network_options(url: str, network_options: Any, max_size_bytes: int, rate_limiter: RateLimiter | None = None) → bytes

Fetch an image applying every field of a NetworkFetchOptions object.

Single place that maps NetworkFetchOptions fields onto fetch parameters, so converters cannot silently drop options (each field added to NetworkFetchOptions must be forwarded here).

Parameters:

url (str) – URL to fetch image from
network_options (Any) – NetworkFetchOptions-like object (see all2md.options.common)
max_size_bytes (int) – Maximum allowed response size in bytes (from BaseParserOptions)
rate_limiter (RateLimiter | None, default None) – Rate limiter shared across a converter’s fetches; build one with RateLimiter.from_options(network_options)

Returns:

Image data

Return type:

bytes

Raises:

NetworkSecurityError – If URL fails security validation or fetch constraints

all2md.utils.network_security.is_network_disabled() → bool

Check if network access is globally disabled via environment variable.

Returns:: True if network access should be disabled, False otherwise
Return type:: bool