all2md.ast.transforms

AST transformation and manipulation utilities.

This module provides visitors and utilities for transforming and manipulating AST structures. It enables filtering nodes, applying transformations, cloning trees, and performing common document processing tasks.

Examples

Extract all headings from a document:

>>> from all2md.ast import transforms
>>> headings = transforms.extract_nodes(doc, Heading)
>>> for heading in headings:
...     print(f"Level {heading.level}: {heading.content}")

Remove all images from a document:

>>> filtered_doc = transforms.filter_nodes(doc, lambda n: not isinstance(n, Image))

Change heading levels:

>>> transformer = transforms.HeadingLevelTransformer(offset=1)
>>> new_doc = transforms.transform_nodes(doc, transformer)

class all2md.ast.transforms.NodeTransformer

Bases: NodeVisitor

Base class for transforming AST nodes.

Subclasses should implement visit_* methods that return modified nodes or None to remove nodes. The transformer creates a new AST with the transformations applied.

Examples

>>> class UppercaseTransformer(NodeTransformer):
...     def visit_text(self, node):
...         return Text(content=node.content.upper())
>>>
>>> transformer = UppercaseTransformer()
>>> new_doc = transformer.transform(doc)

transform(node: Node) → Node | None

Transform an AST node.

Parameters:: node (Node) – Node to transform
Returns:: Transformed node or None to remove
Return type:: Node or None

visit_document(node: Document) → Document: Transform a Document node.

visit_heading(node: Heading) → Heading: Transform a Heading node.

visit_paragraph(node: Paragraph) → Paragraph: Transform a Paragraph node.

visit_code_block(node: CodeBlock) → CodeBlock: Transform a CodeBlock node.

visit_block_quote(node: BlockQuote) → BlockQuote: Transform a BlockQuote node.

visit_list(node: List) → List: Transform a List node.

visit_list_item(node: ListItem) → ListItem: Transform a ListItem node.

visit_table(node: Table) → Table: Transform a Table node.

visit_table_row(node: TableRow) → TableRow: Transform a TableRow node.

visit_table_cell(node: TableCell) → TableCell: Transform a TableCell node.

visit_thematic_break(node: ThematicBreak) → ThematicBreak: Transform a ThematicBreak node.

visit_html_block(node: HTMLBlock) → HTMLBlock: Transform an HTMLBlock node.

visit_comment(node: Comment) → Comment: Transform a Comment node (block-level).

visit_text(node: Text) → Text: Transform a Text node.

visit_emphasis(node: Emphasis) → Emphasis: Transform an Emphasis node.

visit_strong(node: Strong) → Strong: Transform a Strong node.

visit_code(node: Code) → Code: Transform a Code node.

visit_link(node: Link) → Link: Transform a Link node.

visit_image(node: Image) → Image: Transform an Image node.

visit_line_break(node: LineBreak) → LineBreak: Transform a LineBreak node.

visit_strikethrough(node: Strikethrough) → Strikethrough: Transform a Strikethrough node.

visit_mark(node: Mark) → Mark: Transform a Mark (highlight) node.

visit_underline(node: Underline) → Underline: Transform an Underline node.

visit_superscript(node: Superscript) → Superscript: Transform a Superscript node.

visit_subscript(node: Subscript) → Subscript: Transform a Subscript node.

visit_html_inline(node: HTMLInline) → HTMLInline: Transform an HTMLInline node.

visit_comment_inline(node: CommentInline) → CommentInline: Transform a CommentInline node (inline).

visit_footnote_reference(node: FootnoteReference) → FootnoteReference: Transform a FootnoteReference node.

visit_math_inline(node: MathInline) → MathInline: Transform a MathInline node.

visit_footnote_definition(node: FootnoteDefinition) → FootnoteDefinition: Transform a FootnoteDefinition node.

visit_definition_list(node: DefinitionList) → DefinitionList: Transform a DefinitionList node.

visit_definition_term(node: DefinitionTerm) → DefinitionTerm: Transform a DefinitionTerm node.

visit_definition_description(node: DefinitionDescription) → DefinitionDescription: Transform a DefinitionDescription node.

visit_math_block(node: MathBlock) → MathBlock: Transform a MathBlock node.

class all2md.ast.transforms.NodeCollector

Bases: NodeVisitor

Visitor that collects nodes matching a condition.

Parameters:: predicate (callable or None, default = None) – Function that takes a node and returns True to collect it

Initialize the collector with an optional predicate function.

__init__(predicate: Callable[[Node], bool] | None = None): Initialize the collector with an optional predicate function.

visit_document(node: Document) → None: Visit a Document node.

visit_heading(node: Heading) → None: Visit a Heading node.

visit_paragraph(node: Paragraph) → None: Visit a Paragraph node.

visit_code_block(node: CodeBlock) → None: Visit a CodeBlock node.

visit_block_quote(node: BlockQuote) → None: Visit a BlockQuote node.

visit_list(node: List) → None: Visit a List node.

visit_list_item(node: ListItem) → None: Visit a ListItem node.

visit_table(node: Table) → None: Visit a Table node.

visit_table_row(node: TableRow) → None: Visit a TableRow node.

visit_table_cell(node: TableCell) → None: Visit a TableCell node.

visit_thematic_break(node: ThematicBreak) → None: Visit a ThematicBreak node.

visit_html_block(node: HTMLBlock) → None: Visit an HTMLBlock node.

visit_comment(node: Comment) → None: Visit a Comment node (block-level).

visit_text(node: Text) → None: Visit a Text node.

visit_emphasis(node: Emphasis) → None: Visit an Emphasis node.

visit_strong(node: Strong) → None: Visit a Strong node.

visit_code(node: Code) → None: Visit a Code node.

visit_link(node: Link) → None: Visit a Link node.

visit_image(node: Image) → None: Visit an Image node.

visit_line_break(node: LineBreak) → None: Visit a LineBreak node.

visit_strikethrough(node: Strikethrough) → None: Visit a Strikethrough node.

visit_mark(node: Mark) → None: Visit a Mark (highlight) node.

visit_underline(node: Underline) → None: Visit an Underline node.

visit_superscript(node: Superscript) → None: Visit a Superscript node.

visit_subscript(node: Subscript) → None: Visit a Subscript node.

visit_html_inline(node: HTMLInline) → None: Visit an HTMLInline node.

visit_comment_inline(node: CommentInline) → None: Visit a CommentInline node (inline).

visit_footnote_reference(node: FootnoteReference) → None: Visit a FootnoteReference node.

visit_math_inline(node: MathInline) → None: Visit a MathInline node.

visit_footnote_definition(node: FootnoteDefinition) → None: Visit a FootnoteDefinition node.

visit_definition_list(node: DefinitionList) → None: Visit a DefinitionList node.

visit_definition_term(node: DefinitionTerm) → None: Visit a DefinitionTerm node.

visit_definition_description(node: DefinitionDescription) → None: Visit a DefinitionDescription node.

visit_math_block(node: MathBlock) → None: Visit a MathBlock node.

all2md.ast.transforms.clone_node(node: Node) → Node

Create a deep copy of an AST node.

Parameters:: node (Node) – Node to clone
Returns:: Deep copy of the node
Return type:: Node

Examples

>>> cloned_doc = clone_node(doc)
>>> cloned_doc is doc  # False
False

all2md.ast.transforms.extract_nodes(doc: Document, node_type: Type[Node] | None = None) → list[Node]

Extract all nodes of a specific type from a document.

Parameters:

doc (Document) – Document to extract from
node_type (type or None, default = None) – Node type to extract (None for all nodes)

Returns:

All matching nodes

Return type:

list of Node

Examples

>>> headings = extract_nodes(doc, Heading)
>>> images = extract_nodes(doc, Image)

all2md.ast.transforms.filter_nodes(doc: Document, predicate: Callable[[Node], bool]) → Document

Filter nodes from a document based on a condition.

Parameters:

doc (Document) – Document to filter
predicate (callable) – Function that takes a node and returns True to keep it

Returns:

New document with filtered nodes

Return type:

Document

Notes

The root Document node is always preserved, regardless of the predicate. Only the children of the Document are filtered according to the predicate.

Examples

Remove all images:

>>> filtered_doc = filter_nodes(doc, lambda n: not isinstance(n, Image))

Keep only headings and paragraphs:

>>> filtered_doc = filter_nodes(doc, lambda n: isinstance(n, (Heading, Paragraph)))

all2md.ast.transforms.transform_nodes(doc: Document, transformer: NodeTransformer) → Document

Apply a transformation visitor to a document.

Parameters:

doc (Document) – Document to transform
transformer (NodeTransformer) – Transformer to apply

Returns:

Transformed document

Return type:

Document

Examples

>>> transformer = HeadingLevelTransformer(offset=1)
>>> new_doc = transform_nodes(doc, transformer)

all2md.ast.transforms.merge_documents(docs: list[Document], metadata_merger: Callable[[dict[str, Any], dict[str, Any]], dict[str, Any]] | None = None) → Document

Merge multiple documents into a single document.

Parameters:

docs (list of Document) – Documents to merge
metadata_merger (callable or None, default = None) –
Optional function to customize metadata merging. Takes (existing_metadata, new_metadata) and returns merged metadata dict. If None, uses last-write-wins strategy where later documents overwrite earlier ones for duplicate keys.

Common strategies provided: - last_write_wins_merger: Later values overwrite (default behavior) - first_write_wins_merger: Earlier values are preserved - merge_lists_merger: Concatenate list values, last-wins for others

Returns:

Merged document with all children and combined metadata

Return type:

Document

Examples

Basic merge with default last-write-wins:

>>> merged = merge_documents([doc1, doc2, doc3])

Preserve first document’s metadata values:

>>> merged = merge_documents([doc1, doc2], metadata_merger=first_write_wins_merger)

Concatenate list-valued metadata:

>>> merged = merge_documents([doc1, doc2], metadata_merger=merge_lists_merger)

Custom merger:

>>> def custom_merger(existing, new):
...     # Custom logic here
...     return merged_dict
>>> merged = merge_documents([doc1, doc2], metadata_merger=custom_merger)

Notes

The default behavior (when metadata_merger=None) uses last-write-wins strategy, meaning later documents’ metadata values overwrite earlier ones for duplicate keys.

all2md.ast.transforms.last_write_wins_merger(existing: dict[str, Any], new: dict[str, Any]) → dict[str, Any]

Merge metadata with last-write-wins strategy (default).

Later document’s metadata values overwrite earlier ones for duplicate keys.

Parameters:

existing (dict) – Existing accumulated metadata
new (dict) – New metadata to merge in

Returns:

Merged metadata dictionary

Return type:

dict

Examples

>>> existing = {"author": "Alice", "version": "1.0"}
>>> new = {"version": "2.0", "date": "2025-01-01"}
>>> last_write_wins_merger(existing, new)
{"author": "Alice", "version": "2.0", "date": "2025-01-01"}

all2md.ast.transforms.first_write_wins_merger(existing: dict[str, Any], new: dict[str, Any]) → dict[str, Any]

Merge metadata with first-write-wins strategy.

Earlier document’s metadata values are preserved for duplicate keys.

Parameters:

existing (dict) – Existing accumulated metadata
new (dict) – New metadata to merge in

Returns:

Merged metadata dictionary

Return type:

dict

Examples

>>> existing = {"author": "Alice", "version": "1.0"}
>>> new = {"version": "2.0", "date": "2025-01-01"}
>>> first_write_wins_merger(existing, new)
{"author": "Alice", "version": "1.0", "date": "2025-01-01"}

all2md.ast.transforms.merge_lists_merger(existing: dict[str, Any], new: dict[str, Any]) → dict[str, Any]

Merge metadata with list concatenation for list values.

When both existing and new have list values for the same key, concatenate them. For non-list values, uses last-write-wins strategy.

Parameters:

existing (dict) – Existing accumulated metadata
new (dict) – New metadata to merge in

Returns:

Merged metadata dictionary

Return type:

dict

Examples

>>> existing = {"tags": ["python", "ast"], "version": "1.0"}
>>> new = {"tags": ["markdown"], "version": "2.0"}
>>> merge_lists_merger(existing, new)
{"tags": ["python", "ast", "markdown"], "version": "2.0"}

class all2md.ast.transforms.HeadingLevelTransformer

Bases: NodeTransformer

Transformer that adjusts heading levels by an offset.

Parameters:

offset (int) – Amount to shift heading levels (can be negative)
min_level (int, default = 1) – Minimum allowed heading level
max_level (int, default = 6) – Maximum allowed heading level

Examples

>>> # Increase all heading levels by 1
>>> transformer = HeadingLevelTransformer(offset=1)
>>> new_doc = transformer.transform(doc)

Initialize the transform with offset and level constraints.

__init__(offset: int, min_level: int = 1, max_level: int = 6): Initialize the transform with offset and level constraints.

visit_heading(node: Heading) → Heading: Transform heading level.

class all2md.ast.transforms.LinkRewriter

Bases: NodeTransformer

Transformer that rewrites link URLs.

Parameters:

url_mapper (callable) – Function that takes a URL string and returns a new URL string
validate_urls (bool, default = True) – Whether to validate URLs after rewriting for dangerous schemes. When True, raises ValueError if the url_mapper produces a URL with a dangerous scheme (javascript:, vbscript:, etc.). This provides defense-in-depth against accidentally creating unsafe URLs.

Raises:

ValueError – If validate_urls=True and url_mapper produces a URL with a dangerous or unrecognized scheme

Examples

>>> # Convert relative links to absolute
>>> def make_absolute(url):
...     if url.startswith('/'):
...         return f'https://example.com{url}'
...     return url
>>> transformer = LinkRewriter(make_absolute)
>>> new_doc = transformer.transform(doc)

>>> # Disable validation if you need to generate non-standard URLs
>>> transformer = LinkRewriter(my_mapper, validate_urls=False)

Notes

Security Considerations:: By default (validate_urls=True), this transformer validates that the url_mapper does not produce dangerous URLs. This prevents accidental creation of XSS vectors like javascript: or data:text/html URLs. Only disable validation if you have a specific need and understand the security implications.

Initialize the transform with a URL mapping function.

Parameters:

url_mapper (callable) – Function that takes a URL string and returns a new URL string
validate_urls (bool, default = True) – Whether to validate URLs after rewriting for dangerous schemes

__init__(url_mapper: Callable[[str], str], validate_urls: bool = True)

Initialize the transform with a URL mapping function.

Parameters:

url_mapper (callable) – Function that takes a URL string and returns a new URL string
validate_urls (bool, default = True) – Whether to validate URLs after rewriting for dangerous schemes

visit_link(node: Link) → Link: Rewrite link URL.

visit_image(node: Image) → Image: Rewrite image URL.

class all2md.ast.transforms.TextReplacer

Bases: NodeTransformer

Transformer that replaces text patterns.

Parameters:

pattern (str) – Pattern to search for
replacement (str) – Replacement string
use_regex (bool, default = False) – Whether to use regex matching

Raises:

ValueError – If use_regex=True and pattern is not a valid regular expression
SecurityError – If use_regex=True and pattern contains dangerous constructs that could lead to ReDoS (Regular Expression Denial of Service) attacks

Examples

>>> # Replace all occurrences of "foo" with "bar"
>>> transformer = TextReplacer("foo", "bar")
>>> new_doc = transformer.transform(doc)

>>> # Use regex for pattern matching
>>> transformer = TextReplacer(r"\\d+", "NUMBER", use_regex=True)
>>> new_doc = transformer.transform(doc)

Notes

For security reasons, when use_regex=True, this transform validates user-supplied regex patterns to prevent ReDoS attacks. Patterns with nested quantifiers or excessive backtracking potential are rejected. See validate_user_regex_pattern() for details on what patterns are considered safe.

Initialize the transform with pattern and replacement.

Parameters:

pattern (str) – Pattern to search for (literal string or regex)
replacement (str) – Replacement string
use_regex (bool, default = False) – Whether to use regex matching

Raises:

ValueError – If use_regex=True and pattern is not a valid regular expression
SecurityError – If use_regex=True and pattern contains dangerous constructs

__init__(pattern: str, replacement: str, use_regex: bool = False)

Initialize the transform with pattern and replacement.

Parameters:

pattern (str) – Pattern to search for (literal string or regex)
replacement (str) – Replacement string
use_regex (bool, default = False) – Whether to use regex matching

Raises:

ValueError – If use_regex=True and pattern is not a valid regular expression
SecurityError – If use_regex=True and pattern contains dangerous constructs

visit_text(node: Text) → Text: Replace text content.

class all2md.ast.transforms.InlineFormattingConsolidator

Bases: NodeTransformer

Transformer that consolidates fragmented inline formatting nodes.

This transformer fixes common PDF parsing artifacts where inline formatting (bold, italic) is fragmented across multiple adjacent nodes. It performs:

Merges adjacent same-type formatting nodes (Strong+Strong, Emphasis+Emphasis)
Moves trailing/leading whitespace outside formatting markers
Removes empty formatting nodes after whitespace extraction
Merges adjacent Text nodes

Examples

>>> # Fix fragmented bold: **text** **more** -> **text more**
>>> consolidator = InlineFormattingConsolidator()
>>> fixed_doc = consolidator.transform(doc)

>>> # Fix trailing whitespace: **text ** -> **text** + space
>>> consolidator = InlineFormattingConsolidator()
>>> fixed_doc = consolidator.transform(doc)

Notes

This transformer is particularly useful for PDF-to-Markdown conversion where PyMuPDF creates separate text spans at word or formatting boundaries.

visit_paragraph(node: Paragraph) → Paragraph: Consolidate inline formatting in a Paragraph.

visit_heading(node: Heading) → Heading: Consolidate inline formatting in a Heading.

visit_table_cell(node: TableCell) → TableCell: Consolidate inline formatting in a TableCell.

visit_list_item(node: ListItem) → ListItem: Consolidate inline formatting in a ListItem.

visit_block_quote(node: BlockQuote) → BlockQuote: Consolidate inline formatting in BlockQuote children.