all2md.transforms.builtin

Built-in transforms for common document processing tasks.

This module provides a collection of ready-to-use transforms for common document manipulation operations. All transforms are registered via entry points for discovery and CLI usage.

Available Transforms

  • RemoveImagesTransform: Remove all images

  • RemoveNodesTransform: Remove nodes of specified types

  • HeadingOffsetTransform: Shift heading levels

  • TitlePromotionTransform: Promote leading H1 to title and shift subsequent headings

  • LinkRewriterTransform: Rewrite URLs with regex patterns

  • TextReplacerTransform: Find and replace text

  • AddHeadingIdsTransform: Generate unique IDs for headings

  • RemoveBoilerplateTextTransform: Remove common boilerplate patterns

  • AddConversionTimestampTransform: Add timestamp to metadata

  • CalculateWordCountTransform: Calculate word and character counts

  • AddAttachmentFootnotesTransform: Add footnote definitions for attachment references

  • GenerateTocTransform: Generate table of contents from document headings

Examples

Remove all images:

>>> transform = RemoveImagesTransform()
>>> new_doc = transform.transform(doc)

Offset headings by 2 levels:

>>> transform = HeadingOffsetTransform(offset=2)
>>> new_doc = transform.transform(doc)

Add unique IDs to headings:

>>> transform = AddHeadingIdsTransform(id_prefix="doc-")
>>> new_doc = transform.transform(doc)

Add footnote definitions for attachment references:

>>> transform = AddAttachmentFootnotesTransform(section_title="Image Sources")
>>> new_doc = transform.transform(doc)

Generate table of contents:

>>> transform = GenerateTocTransform(max_depth=3, position="top")
>>> new_doc = transform.transform(doc)
class all2md.transforms.builtin.RemoveImagesTransform

Bases: NodeTransformer

Remove all Image nodes from the AST.

This transform removes every Image node it encounters, useful for creating text-only versions of documents or reducing document size.

Examples

>>> transform = RemoveImagesTransform()
>>> doc_without_images = transform.transform(document)
visit_image(node: Image) → None

Remove image by returning None.

Parameters:

node (Image) – Image node to remove

Returns:

Always returns None to remove the node

Return type:

None

class all2md.transforms.builtin.RemoveNodesTransform

Bases: NodeTransformer

Remove nodes of specified types from the AST.

This is a generic transform that can remove any combination of node types. Useful for stripping specific elements like tables, code blocks, or any other node type.

Parameters:

node_types (list[str]) – List of node type names to remove (e.g., [‘image’, ‘table’, ‘code_block’])

Examples

Remove images and tables:

>>> transform = RemoveNodesTransform(node_types=['image', 'table'])
>>> cleaned_doc = transform.transform(document)

Initialize with list of node types to remove.

Parameters:

node_types (list[str]) – Node type names to remove

Raises:

ValueError – If ‘document’ is in node_types (cannot remove root node), or if any node_type is unknown (typo detection)

__init__(node_types: list[str])

Initialize with list of node types to remove.

Parameters:

node_types (list[str]) – Node type names to remove

Raises:

ValueError – If ‘document’ is in node_types (cannot remove root node), or if any node_type is unknown (typo detection)

transform(node: Node) → Node | None

Transform node, removing it if it matches specified types.

Parameters:

node (Node) – Node to potentially remove

Returns:

None if node should be removed, otherwise transformed node

Return type:

Node or None

class all2md.transforms.builtin.HeadingOffsetTransform

Bases: NodeTransformer

Shift heading levels by a specified offset.

This transform adjusts all heading levels in the document by adding an offset value. Levels are clamped to the valid range of 1-6.

Parameters:

offset (int, default = 1) – Number of levels to shift (positive to increase, negative to decrease)

Examples

Increase all heading levels by 1 (H1 becomes H2):

>>> transform = HeadingOffsetTransform(offset=1)
>>> new_doc = transform.transform(document)

Decrease all heading levels by 1 (H2 becomes H1):

>>> transform = HeadingOffsetTransform(offset=-1)
>>> new_doc = transform.transform(document)

Initialize with heading level offset.

Parameters:

offset (int) – Heading level adjustment

__init__(offset: int = 1)

Initialize with heading level offset.

Parameters:

offset (int) – Heading level adjustment

visit_heading(node: Heading) → Heading

Adjust heading level.

Parameters:

node (Heading) – Heading node to adjust

Returns:

Heading with adjusted level

Return type:

Heading

class all2md.transforms.builtin.TitlePromotionTransform

Bases: NodeTransformer

Promote a leading H1 to a document title and shift subsequent headings.

When converting Markdown to Word, a leading # Heading is typically the document title rather than a “Heading 1”. This transform detects a leading H1 (skipping empty / whitespace-only paragraphs before it), marks it with metadata["is_title"] = True, and promotes all subsequent headings by one level (H2 → H1, H3 → H2, etc.) so they style properly under the title.

If the first real content node is not an H1, the document passes through unchanged.

Examples

>>> transform = TitlePromotionTransform()
>>> new_doc = transform.transform(document)
transform(node: Node) → Node

Apply title promotion to the document.

Parameters:

node (Node) – Root node (expected to be a Document)

Returns:

Document with title metadata and promoted headings

Return type:

Node

class all2md.transforms.builtin.LinkRewriterTransform

Bases: NodeTransformer

Rewrite link URLs using regex pattern matching.

This transform allows flexible URL rewriting using regular expressions. Useful for converting relative links to absolute, updating base URLs, or modifying link schemes.

Parameters:

  • pattern (str) – Regex pattern to match in URLs

  • replacement (str) – Replacement string (can include regex groups like \1, \2)

Raises:

SecurityError – If the pattern contains dangerous constructs that could lead to ReDoS (Regular Expression Denial of Service) attacks

Examples

Convert relative links to absolute:

>>> transform = LinkRewriterTransform(
...     pattern=r'^/docs/',
...     replacement='https://example.com/docs/'
... )
>>> new_doc = transform.transform(document)

Notes

For security reasons, this transform validates user-supplied regex patterns to prevent ReDoS attacks. Patterns with nested quantifiers or excessive backtracking potential are rejected. See validate_user_regex_pattern() for details on what patterns are considered safe.

Initialize with pattern and replacement.

Parameters:

  • pattern (str) – Regex pattern

  • replacement (str) – Replacement string

Raises:

SecurityError – If pattern contains dangerous constructs

__init__(pattern: str, replacement: str)

Initialize with pattern and replacement.

Parameters:

  • pattern (str) – Regex pattern

  • replacement (str) – Replacement string

Raises:

SecurityError – If pattern contains dangerous constructs

Rewrite link URL if it matches pattern.

Parameters:

node (Link) – Link node to potentially rewrite

Returns:

Link with potentially rewritten URL

Return type:

Link

class all2md.transforms.builtin.TextReplacerTransform

Bases: NodeTransformer

Find and replace text in Text nodes.

This transform performs simple text replacement across all Text nodes in the document. For regex-based replacement, use a custom transform.

Parameters:

  • find (str) – Text to find

  • replace (str) – Replacement text

Examples

Replace all instances of “TODO”:

>>> transform = TextReplacerTransform(find="TODO", replace="DONE")
>>> new_doc = transform.transform(document)

Initialize with find and replace strings.

Parameters:

  • find (str) – Text to find

  • replace (str) – Replacement text

__init__(find: str, replace: str)

Initialize with find and replace strings.

Parameters:

  • find (str) – Text to find

  • replace (str) – Replacement text

visit_text(node: Text) → Text

Replace text in node content.

Parameters:

node (Text) – Text node to process

Returns:

Text node with replacements applied

Return type:

Text

class all2md.transforms.builtin.AddHeadingIdsTransform

Bases: NodeTransformer

Generate and add unique IDs to heading nodes.

This transform creates slugified IDs from heading text and adds them to the heading metadata. These IDs can be used by renderers to create HTML anchors for linkable sections.

Parameters:

  • id_prefix (str, default = "") – Prefix to add to all generated IDs

  • separator (str, default = "-") – Separator for multi-word slugs and duplicate handling

Examples

Basic usage:

>>> transform = AddHeadingIdsTransform()
>>> new_doc = transform.transform(document)
>>> # "My Heading" -> metadata['id'] = "my-heading"

With prefix:

>>> transform = AddHeadingIdsTransform(id_prefix="doc-")
>>> new_doc = transform.transform(document)
>>> # "My Heading" -> metadata['id'] = "doc-my-heading"

Initialize with prefix and separator.

Parameters:

  • id_prefix (str) – Prefix for IDs

  • separator (str) – Word separator

__init__(id_prefix: str = '', separator: str = '-')

Initialize with prefix and separator.

Parameters:

  • id_prefix (str) – Prefix for IDs

  • separator (str) – Word separator

visit_heading(node: Heading) → Heading

Add unique ID to heading.

Parameters:

node (Heading) – Heading to process

Returns:

Heading with ID in metadata

Return type:

Heading

class all2md.transforms.builtin.RemoveBoilerplateTextTransform

Bases: NodeTransformer

Remove paragraphs matching common boilerplate patterns.

This transform removes paragraphs whose text matches predefined patterns like “CONFIDENTIAL”, “Page X of Y”, etc. Useful for cleaning up corporate documents and reports.

Parameters:

  • patterns (list[str], optional) – List of regex patterns to match (default: common boilerplate)

  • skip_if_truncated (bool, default = True) – If True, skip pattern matching when text exceeds MAX_TEXT_LENGTH_FOR_REGEX to avoid false positives with end-anchored patterns ($). If False, match against truncated text (may produce incorrect results with anchors).

Raises:

SecurityError – If any user-supplied pattern contains dangerous constructs that could lead to ReDoS (Regular Expression Denial of Service) attacks

Examples

Use default patterns:

>>> transform = RemoveBoilerplateTextTransform()
>>> cleaned_doc = transform.transform(document)

Custom patterns with anchoring:

>>> transform = RemoveBoilerplateTextTransform(
...     patterns=[r"^DRAFT$", r"^INTERNAL ONLY$", r"^Page \d+ of \d+$"]
... )
>>> cleaned_doc = transform.transform(document)

Allow matching truncated text (not recommended):

>>> transform = RemoveBoilerplateTextTransform(skip_if_truncated=False)
>>> cleaned_doc = transform.transform(document)

Notes

Pattern Matching Semantics: This transform uses Python’s re.match(), which implicitly anchors at the start of the string (equivalent to adding ^ at the beginning). For exact matching of entire paragraphs, patterns should include an end anchor ($). For example:

  • r”CONFIDENTIAL” - Matches paragraphs starting with “CONFIDENTIAL”

  • r”CONFIDENTIAL$” - Matches paragraphs that are exactly “CONFIDENTIAL” or start with “CONFIDENTIAL” followed by only whitespace

  • r”^CONFIDENTIAL$” - Explicitly anchored (redundant ^, but clearer)

If you need to match patterns anywhere in the text (not just at the start), use re.search() semantics by implementing a custom transform.

Security: For security reasons, this transform validates user-supplied regex patterns to prevent ReDoS attacks. Default patterns are pre-validated and trusted. Patterns with nested quantifiers or excessive backtracking potential are rejected. See validate_user_regex_pattern() for details.

Truncation Behavior: Text longer than MAX_TEXT_LENGTH_FOR_REGEX (10000 characters) is truncated before matching for ReDoS protection. With skip_if_truncated=True (default), such paragraphs are preserved to avoid false positives from patterns using end anchors ($). This is safer but may miss some boilerplate. With skip_if_truncated=False, matching proceeds on truncated text, which may incorrectly match or not match patterns with anchors.

Initialize with patterns.

Parameters:

  • patterns (list[str] or None) – Regex patterns to match (None uses defaults)

  • skip_if_truncated (bool) – Skip matching when text is truncated (safer default)

Raises:

SecurityError – If any user-supplied pattern contains dangerous constructs

__init__(patterns: list[str] | None = None, skip_if_truncated: bool = True)

Initialize with patterns.

Parameters:

  • patterns (list[str] or None) – Regex patterns to match (None uses defaults)

  • skip_if_truncated (bool) – Skip matching when text is truncated (safer default)

Raises:

SecurityError – If any user-supplied pattern contains dangerous constructs

visit_paragraph(node: Paragraph) → Paragraph | None

Remove paragraph if it matches boilerplate pattern.

Parameters:

node (Paragraph) – Paragraph to check

Returns:

None if matches boilerplate, otherwise paragraph

Return type:

Paragraph or None

class all2md.transforms.builtin.AddConversionTimestampTransform

Bases: NodeTransformer

Add conversion timestamp to document metadata.

This transform adds a timezone-aware UTC timestamp to the document metadata indicating when the conversion occurred. Useful for tracking document versions and conversion history. All timestamps are generated in UTC to ensure consistency across different time zones.

Parameters:

  • field_name (str, default = "conversion_timestamp") – Metadata field name for the timestamp

  • timestamp_format (str, default = "iso") – Timestamp format: “iso” for ISO 8601 with timezone, “unix” for Unix timestamp, or any strftime format string

  • timespec (str, default = "seconds") – Time precision for ISO format timestamps. Valid values are: - “auto”: Automatic precision - “hours”: Hours precision - “minutes”: Minutes precision - “seconds”: Seconds precision (default, reduces noisy diffs) - “milliseconds”: Milliseconds precision - “microseconds”: Microseconds precision Only applies when timestamp_format=”iso”. Ignored for other formats.

Examples

Add ISO 8601 timestamp with second precision (default):

>>> transform = AddConversionTimestampTransform()
>>> new_doc = transform.transform(document)
>>> # metadata['conversion_timestamp'] = "2025-01-01T12:00:00+00:00"

Add ISO 8601 timestamp with microsecond precision:

>>> transform = AddConversionTimestampTransform(timespec="microseconds")
>>> new_doc = transform.transform(document)
>>> # metadata['conversion_timestamp'] = "2025-01-01T12:00:00.123456+00:00"

Add Unix timestamp:

>>> transform = AddConversionTimestampTransform(timestamp_format="unix")
>>> new_doc = transform.transform(document)
>>> # metadata['conversion_timestamp'] = "1735732800"

Custom strftime format:

>>> transform = AddConversionTimestampTransform(
...     field_name="converted_at",
...     timestamp_format="%Y-%m-%d %H:%M:%S UTC"
... )
>>> new_doc = transform.transform(document)
>>> # metadata['converted_at'] = "2025-01-01 12:00:00 UTC"

Notes

All timestamps are generated in UTC (Coordinated Universal Time) using datetime.now(timezone.utc). This ensures consistent timestamps regardless of the server’s local timezone.

The default timespec=”seconds” is recommended to reduce noisy git diffs when regenerating documents, as subsecond precision is rarely needed for document conversion timestamps.

Initialize with field name, format, and time precision.

Parameters:

  • field_name (str) – Metadata field name

  • timestamp_format (str) – Timestamp format

  • timespec (str) – Time precision for ISO format (default: “seconds”)

__init__(field_name: str = 'conversion_timestamp', timestamp_format: str = 'iso', timespec: str = 'seconds')

Initialize with field name, format, and time precision.

Parameters:

  • field_name (str) – Metadata field name

  • timestamp_format (str) – Timestamp format

  • timespec (str) – Time precision for ISO format (default: “seconds”)

visit_document(node: Document) → Document

Add timestamp to document metadata.

Parameters:

node (Document) – Document to process

Returns:

Document with timestamp in metadata

Return type:

Document

class all2md.transforms.builtin.CalculateWordCountTransform

Bases: NodeTransformer

Calculate word and character counts and add to metadata.

This transform traverses the entire document, extracts all text, and calculates word and character counts. The counts are added to the document metadata.

Parameters:

  • word_field (str, default = "word_count") – Metadata field name for word count

  • char_field (str, default = "char_count") – Metadata field name for character count

Examples

Basic usage:

>>> transform = CalculateWordCountTransform()
>>> new_doc = transform.transform(document)
>>> # metadata['word_count'] = 150
>>> # metadata['char_count'] = 890

Custom field names:

>>> transform = CalculateWordCountTransform(
...     word_field="words",
...     char_field="characters"
... )
>>> new_doc = transform.transform(document)

Notes

Character Count Behavior: The char_count metric represents the length of normalized text extracted from the AST, not the original document’s character count. During text extraction, text fragments from separate AST nodes are joined with spaces, which may introduce synthetic spacing not present in the original document. For example, if the AST contains two adjacent Text nodes Text("hello") and Text("world"), the extracted text will be "hello world" (11 characters including the inserted space), even though the original text nodes only contain 10 characters total.

This normalized approach provides consistent metrics across different AST structures, though it may not exactly match the original document’s byte count. Word count is calculated by splitting the normalized text on whitespace, which is generally more robust to these variations.

Initialize with field names.

Parameters:

  • word_field (str) – Field name for word count

  • char_field (str) – Field name for character count

__init__(word_field: str = 'word_count', char_field: str = 'char_count')

Initialize with field names.

Parameters:

  • word_field (str) – Field name for word count

  • char_field (str) – Field name for character count

visit_document(node: Document) → Document

Calculate counts and add to metadata.

Parameters:

node (Document) – Document to analyze

Returns:

Document with counts in metadata

Return type:

Document

class all2md.transforms.builtin.AddAttachmentFootnotesTransform

Bases: NodeTransformer

Add footnote definitions for attachment references.

When attachments are processed with alt_text_mode=”footnote”, they generate footnote-style references like ![image][^label] but no corresponding definitions. This transform scans the AST for such references and adds FootnoteDefinition nodes with source information.

Parameters:

  • section_title (str or None, default "Attachments") – Title for the footnote section heading. If None, no heading is added.

  • add_definitions_for_images (bool, default True) – Add definitions for image footnote references

  • add_definitions_for_links (bool, default True) – Add definitions for link footnote references

Examples

Add footnote definitions after conversion:

>>> transform = AddAttachmentFootnotesTransform()
>>> doc_with_footnotes = transform.transform(document)

Custom section title:

>>> transform = AddAttachmentFootnotesTransform(section_title="Image Sources")
>>> doc_with_footnotes = transform.transform(document)

Notes

This transform works by: 1. Collecting all Image and Link nodes with empty URLs (indicates footnote mode) 2. Extracting footnote labels from alt text or title 3. Handling duplicate labels by appending numeric suffixes (-2, -3, etc.) 4. Creating FootnoteDefinition nodes with source information 5. Appending definitions to the end of the document

Duplicate labels are resolved using a counter mechanism similar to heading ID generation. When a label appears multiple times, subsequent occurrences get a numeric suffix to ensure unique footnote identifiers

Initialize transform with options.

Parameters:

  • section_title (str or None) – Heading for footnotes section

  • add_definitions_for_images (bool) – Whether to process image footnotes

  • add_definitions_for_links (bool) – Whether to process link footnotes

__init__(section_title: str | None = 'Attachments', add_definitions_for_images: bool = True, add_definitions_for_links: bool = True)

Initialize transform with options.

Parameters:

  • section_title (str or None) – Heading for footnotes section

  • add_definitions_for_images (bool) – Whether to process image footnotes

  • add_definitions_for_links (bool) – Whether to process link footnotes

visit_document(node: Document) → Document

Process document and add footnote definitions.

Parameters:

node (Document) – Document to process

Returns:

Document with footnote definitions added

Return type:

Document

class all2md.transforms.builtin.GenerateTocTransform

Bases: NodeTransformer

Generate a table of contents from document headings.

This transform extracts headings from the document and generates a nested list representing the table of contents. The TOC can be placed at the top or bottom of the document.

Parameters:

  • title (str, default = "Table of Contents") – Title for the TOC section

  • max_depth (int, default = 3) – Maximum heading level to include (1-6)

  • position ({"top", "bottom"}, default = "top") – Position to insert the TOC

  • add_links (bool, default = True) – Whether to create links to headings (requires heading IDs)

  • separator (str, default = "-") – Separator for generating heading IDs when not present

  • set_ids_if_missing (bool, default = False) – If True, inject generated IDs into heading metadata when missing. This ensures renderers create anchors matching the TOC links. If False (default), IDs are only used for TOC links.

Examples

Basic usage:

>>> transform = GenerateTocTransform()
>>> doc_with_toc = transform.transform(document)

Custom depth and position:

>>> transform = GenerateTocTransform(
...     title="Contents",
...     max_depth=2,
...     position="bottom"
... )
>>> doc_with_toc = transform.transform(document)

Inject IDs into headings:

>>> transform = GenerateTocTransform(set_ids_if_missing=True)
>>> doc_with_toc = transform.transform(document)
>>> # Headings now have 'id' in metadata for renderer anchors

Notes

This transform works best when combined with AddHeadingIdsTransform, which generates unique IDs for headings that can be used for navigation. If headings don’t have IDs, the transform will generate slugified IDs on-the-fly for link targets.

ID Injection: With set_ids_if_missing=True, generated IDs are injected into heading metadata so renderers can create matching anchors. This is recommended when not using AddHeadingIdsTransform. Alternatively, run AddHeadingIdsTransform before GenerateTocTransform to ensure all headings have IDs upfront.

Initialize with TOC generation options.

Parameters:

  • title (str) – TOC section title

  • max_depth (int) – Maximum heading level (1-6)

  • position (str) – Position for TOC (“top” or “bottom”)

  • add_links (bool) – Whether to generate links

  • separator (str) – Separator for ID generation

  • set_ids_if_missing (bool) – Inject generated IDs into heading metadata

Raises:

ValueError – If max_depth is not between 1 and 6, or position is invalid

__init__(title: str = 'Table of Contents', max_depth: int = 3, position: str = 'top', add_links: bool = True, separator: str = '-', set_ids_if_missing: bool = False)

Initialize with TOC generation options.

Parameters:

  • title (str) – TOC section title

  • max_depth (int) – Maximum heading level (1-6)

  • position (str) – Position for TOC (“top” or “bottom”)

  • add_links (bool) – Whether to generate links

  • separator (str) – Separator for ID generation

  • set_ids_if_missing (bool) – Inject generated IDs into heading metadata

Raises:

ValueError – If max_depth is not between 1 and 6, or position is invalid

visit_document(node: Document) → Document

Generate TOC and add to document.

Parameters:

node (Document) – Document to process

Returns:

Document with TOC added

Return type:

Document