all2md.utils.input_sources

Abstractions for loading document sources from various locations.

This module defines the loader infrastructure that resolves user-supplied inputs (paths, URLs, streams, etc.) into a consistent representation that parsers can consume. The goal is to make it easy to add new retrieval mechanisms (HTTP(S), cloud object stores, etc.) without touching parser implementations.

class all2md.utils.input_sources.NamedBytesIO

Bases: BytesIO

BytesIO variant that carries a display name for downstream consumers.

Initialize a named bytes IO stream.

Parameters:

initial_bytes (bytes) – Initial byte content
name (str or None, optional) – Display name for the stream

__init__(initial_bytes: bytes, name: str | None = None) → None

Initialize a named bytes IO stream.

Parameters:

initial_bytes (bytes) – Initial byte content
name (str or None, optional) – Display name for the stream

property name: str: Best-effort filename used for format detection and logging.

class all2md.utils.input_sources.RemoteInputOptions

Bases: CloneFrozenMixin

Global options controlling remote document retrieval.

allow_remote_input: bool = False

allowed_hosts: list[str] | None = None

require_https: bool = True

require_head_success: bool = True

timeout: float = 10.0

max_size_bytes: int = 20971520

user_agent: str = 'all2md-fetcher/1.0'

follow_robots_txt: Literal['strict', 'warn', 'ignore'] | str = 'strict'

__init__(allow_remote_input: bool = False, allowed_hosts: list[str] | None = None, require_https: bool = True, require_head_success: bool = True, timeout: float = 10.0, max_size_bytes: int = 20971520, user_agent: str = 'all2md-fetcher/1.0', follow_robots_txt: Literal['strict', 'warn', 'ignore'] | str = 'strict') → None

class all2md.utils.input_sources.DocumentSourceRequest

Bases: object

Encapsulates the input the user provided along with loader options.

raw_input: str | Path | IO[bytes] | IO[str] | bytes

remote_options: RemoteInputOptions | None = None

metadata: dict[str, Any]

progress_callback: Callable[[ProgressEvent], None] | None = None

scheme() → str | None: Best-effort scheme detection for string inputs.

__init__(raw_input: str | ~pathlib.Path | ~typing.IO[bytes] | ~typing.IO[str] | bytes, remote_options: ~all2md.utils.input_sources.RemoteInputOptions | None = None, metadata: dict[str, ~typing.Any] = <factory>, progress_callback: ~typing.Callable[[~all2md.progress.ProgressEvent], None] | None = None) → None

class all2md.utils.input_sources.DocumentSource

Bases: object

Resolved document payload returned by retrievers.

payload: str | Path | IO[bytes] | IO[str] | bytes

display_name: str

origin_uri: str | None = None

as_path() → Path | None: Return payload as Path when available.

__init__(payload: str | Path | IO[bytes] | IO[str] | bytes, display_name: str, origin_uri: str | None = None) → None

class all2md.utils.input_sources.DocumentSourceRetriever

Bases: ABC

Base class for individual source retrievers.

priority: int = 0

abstractmethod can_handle(request: DocumentSourceRequest) → bool: Return True if this retriever can resolve the request.

abstractmethod load(request: DocumentSourceRequest) → DocumentSource

Load the document and return a resolved source.

Parameters:: request (DocumentSourceRequest) – The request containing input source and optional progress callback
Returns:: Resolved document source with payload
Return type:: DocumentSource

Notes

Retrievers should emit progress events via request.progress_callback when performing long-running operations like network downloads.

class all2md.utils.input_sources.DocumentSourceLoader

Bases: object

Coordinator that dispatches requests to registered retrievers.

retrievers: Sequence[DocumentSourceRetriever]

load(request: DocumentSourceRequest) → DocumentSource: Resolve a request by delegating to the first capable retriever.

__init__(retrievers: Sequence[DocumentSourceRetriever]) → None

class all2md.utils.input_sources.LocalPathRetriever

Bases: DocumentSourceRetriever

Resolver for local filesystem paths.

Notes

file:// URL scheme is not currently supported. Use direct filesystem paths (e.g., “/path/to/file” instead of “file:///path/to/file”).

priority: int = 100

can_handle(request: DocumentSourceRequest) → bool

Check if request can be handled as a local file path.

Parameters:: request (DocumentSourceRequest) – The request to check
Returns:: True if the request is a local file path, False otherwise
Return type:: bool

load(request: DocumentSourceRequest) → DocumentSource

Load a document from a local file path.

Parameters:: request (DocumentSourceRequest) – The request containing the file path
Returns:: The loaded document source
Return type:: DocumentSource
Raises:: ValidationError – If the path doesn’t exist or is not a file

class all2md.utils.input_sources.HttpRetriever

Bases: DocumentSourceRetriever

Resolver that downloads documents from HTTP(S) URLs.

priority: int = 80

can_handle(request: DocumentSourceRequest) → bool

Check if request can be handled as an HTTP(S) URL.

Parameters:: request (DocumentSourceRequest) – The request to check
Returns:: True if the request is an HTTP(S) URL, False otherwise
Return type:: bool

load(request: DocumentSourceRequest) → DocumentSource

Load a document from an HTTP(S) URL.

Parameters:

request (DocumentSourceRequest) – The request containing the URL

Returns:

The loaded document source

Return type:

DocumentSource

Raises:

ValidationError – If network access is disabled or remote input is not allowed
NetworkSecurityError – If the request fails security checks
DependencyError – If httpx is not installed

class all2md.utils.input_sources.BytesRetriever

Bases: DocumentSourceRetriever

Resolver for raw byte strings.

priority: int = 50

can_handle(request: DocumentSourceRequest) → bool

Check if request can be handled as raw bytes.

Parameters:: request (DocumentSourceRequest) – The request to check
Returns:: True if the request is raw bytes, False otherwise
Return type:: bool

load(request: DocumentSourceRequest) → DocumentSource

Load a document from raw bytes.

Parameters:: request (DocumentSourceRequest) – The request containing raw bytes
Returns:: The loaded document source
Return type:: DocumentSource

class all2md.utils.input_sources.FileObjectRetriever

Bases: DocumentSourceRetriever

Resolver for existing file-like objects.

priority: int = 40

can_handle(request: DocumentSourceRequest) → bool

Check if request can be handled as a file-like object.

Parameters:: request (DocumentSourceRequest) – The request to check
Returns:: True if the request is a file-like object, False otherwise
Return type:: bool

load(request: DocumentSourceRequest) → DocumentSource

Load a document from a file-like object.

Parameters:: request (DocumentSourceRequest) – The request containing the file-like object
Returns:: The loaded document source
Return type:: DocumentSource

class all2md.utils.input_sources.TextContentRetriever

Bases: DocumentSourceRetriever

Resolver for inline string content.

priority: int = 30

can_handle(request: DocumentSourceRequest) → bool

Check if request can be handled as inline text content.

Parameters:: request (DocumentSourceRequest) – The request to check
Returns:: True if the request is inline text, False otherwise
Return type:: bool

load(request: DocumentSourceRequest) → DocumentSource

Load a document from inline text content.

Parameters:: request (DocumentSourceRequest) – The request containing inline text
Returns:: The loaded document source
Return type:: DocumentSource

all2md.utils.input_sources.default_loader(additional_retrievers: Iterable[DocumentSourceRetriever] | None = None) → DocumentSourceLoader: Create a loader with built-in retrievers and optional extensions.