Plugin Development Guide

Overview

The all2md library supports a plugin system that allows third-party developers to add support for additional document formats without modifying the core library. This system uses Python entry points to automatically discover and register converter plugins.

Plugin Architecture

The plugin system is built around two key components:

  1. ConverterMetadata: A data class that describes the converter’s capabilities

  2. Entry Points: Python packaging mechanism for plugin discovery

When all2md starts up, it automatically scans for plugins registered under the all2md.converters entry point group and loads their metadata. This enables seamless integration of custom formats.

Creating a Plugin

Basic Plugin Structure

A typical plugin package should have the following structure:

all2md_myformat/
├── pyproject.toml          # Package configuration with entry point
├── README.md               # Documentation
├── LICENSE                 # License file
└── src/
    └── all2md_myformat/
        ├── __init__.py     # Metadata registration
        ├── parser.py       # Parser implementation
        ├── renderer.py     # Renderer implementation (optional)
        └── options.py      # Configuration options

Complete Plugin Walkthrough

The best way to learn plugin development is to study a complete, working example. The simpledoc-plugin in the examples/ directory demonstrates all aspects of building a bidirectional converter plugin.

SimpleDoc is a lightweight markup format created specifically for this example. It supports:

  • Frontmatter metadata (YAML-style between --- delimiters)

  • Headings (lines starting with @@)

  • Lists (lines starting with -)

  • Code blocks (triple backticks)

  • Paragraphs (separated by blank lines)

The complete simpledoc-plugin source is available at: examples/plugins/simpledoc-plugin/

Parser Implementation

A parser converts your format into the all2md AST (Abstract Syntax Tree). Here’s how the SimpleDoc parser implements the core parse() method:

SimpleDocParser.parse() - Main parsing entry point
 1    def parse(self, input_data: Union[str, Path, IO[bytes], bytes]) -> Document:
 2        """Parse SimpleDoc input into an AST Document.
 3
 4        Parameters
 5        ----------
 6        input_data : str, Path, IO[bytes], or bytes
 7            The input SimpleDoc document to parse
 8
 9        Returns
10        -------
11        Document
12            AST Document node representing the parsed document structure
13
14        Raises
15        ------
16        ParsingError
17            If parsing fails due to invalid format or corruption
18
19        """
20        # Progress tracking: Emit "started" event so CLI/UI can show conversion progress
21        # For more complex parsers, you might emit progress at multiple stages
22        self._emit_progress("started", "Converting SimpleDoc document", current=0, total=1)
23
24        # Read content: Handle all input types (file path, bytes, IO) uniformly
25        try:
26            content = self._read_content(input_data)
27        except Exception as e:
28            raise ParsingError(f"Failed to read SimpleDoc file: {e}") from e
29
30        try:
31            # Two-phase parsing: First extract metadata, then parse content
32            # This separation keeps parsing logic clean and allows metadata-only extraction
33            metadata, content_text = self._extract_frontmatter(content)
34
35            # Parse content into AST nodes (the core conversion logic)
36            children = self._parse_content(content_text)
37
38            # Progress tracking: Emit "finished" event when conversion is complete
39            self._emit_progress("finished", "SimpleDoc conversion completed", current=1, total=1)
40
41            # Return Document with both children and metadata
42            # Metadata is stored as dict for serialization compatibility
43            return Document(children=children, metadata=metadata.to_dict())
44
45        except ParsingError:
46            raise
47        except Exception as e:
48            raise ParsingError(
49                f"Failed to parse SimpleDoc content: {e}", parsing_stage="content_parsing", original_error=e
50            ) from e

Key parser patterns demonstrated:

  1. Progress tracking: Use _emit_progress() for CLI feedback

  2. Two-phase parsing: Separate metadata extraction from content parsing

  3. Error handling: Wrap in try/except and raise ParsingError

  4. Input flexibility: Handle str, Path, IO[bytes], and bytes

Input Type Handling

The parser must handle multiple input types uniformly:

SimpleDocParser._read_content() - Multi-format input handling
 1    def _read_content(self, input_data: Union[str, Path, IO[bytes], bytes]) -> str:
 2        """Read content from various input types.
 3
 4        Parameters
 5        ----------
 6        input_data : various types
 7            Input data in different formats
 8
 9        Returns
10        -------
11        str
12            Text content
13
14        """
15        # Input flexibility: Support multiple input types for better usability
16        # BaseParser contract requires handling str, Path, IO[bytes], and bytes
17        if isinstance(input_data, (str, Path)):
18            # File path: Read from disk with UTF-8 encoding
19            # Use errors="replace" to handle encoding issues gracefully
20            with open(input_data, "r", encoding="utf-8", errors="replace") as f:
21                return f.read()
22        elif isinstance(input_data, bytes):
23            # Raw bytes: Decode directly (e.g., from memory or network)
24            return input_data.decode("utf-8", errors="replace")
25        elif hasattr(input_data, "read"):
26            # File-like object: Handle both binary and text streams
27            # This supports stdin, BytesIO, StringIO, etc.
28            raw_content = input_data.read()
29            if isinstance(raw_content, bytes):
30                return raw_content.decode("utf-8", errors="replace")
31            return str(raw_content)
32        else:
33            raise ValueError(f"Unsupported input type: {type(input_data)}")

Content Parsing

The core parsing logic uses a line-by-line state machine approach:

SimpleDocParser._parse_content() - State machine parser
 1        remaining_content = "\n".join(lines[end_index + 1 :])
 2        return metadata, remaining_content
 3
 4    def _parse_content(self, content: str) -> list[Node]:
 5        """Parse SimpleDoc content into AST nodes.
 6
 7        Parameters
 8        ----------
 9        content : str
10            Content to parse (without frontmatter)
11
12        Returns
13        -------
14        list[Node]
15            List of AST nodes
16
17        """
18        # State machine parser: Process line by line with look-ahead
19        # This approach is simple and works well for line-based formats
20        children: list[Node] = []
21        lines = content.split("\n")
22        i = 0
23
24        while i < len(lines):
25            line = lines[i]
26
27            # Skip blank lines between blocks
28            if not line.strip():
29                i += 1
30                continue
31
32            # Pattern matching: Check line prefix to determine block type
33            # Order matters: Check most specific patterns first
34
35            # Heading detection: Lines starting with @@
36            if line.strip().startswith("@@"):
37                heading_text = line.strip()[2:].strip()  # Remove marker
38                children.append(Heading(level=1, content=[Text(content=heading_text)]))
39                i += 1
40                continue
41
42            # Code block detection: Triple backticks (option-driven)
43            if self.options.parse_code_blocks and line.strip().startswith("```"):
44                code_content, lines_consumed = self._parse_code_block(lines[i:])
45                if code_content is not None:
46                    children.append(code_content)
47                    i += lines_consumed  # Skip consumed lines
48                    continue
49
50            # List detection: Lines starting with dash (option-driven)
51            if self.options.parse_lists and line.strip().startswith("-"):
52                list_node, lines_consumed = self._parse_list(lines[i:])
53                children.append(list_node)
54                i += lines_consumed  # Skip consumed lines
55                continue
56
57            # Default: Treat as paragraph (multi-line until blank line)
58            para_node, lines_consumed = self._parse_paragraph(lines[i:])
59            if para_node:
60                children.append(para_node)

The parser demonstrates:

  • Pattern matching: Check line prefixes to determine block type

  • Look-ahead parsing: Pass remaining lines to sub-parsers

  • Option-driven behavior: Respect user configuration

  • Clean separation: Each block type has its own parser method

Metadata Extraction

Metadata can be extracted separately from full parsing:

SimpleDocParser._extract_frontmatter() - Metadata parsing
 1    def _extract_frontmatter(self, content: str) -> tuple[DocumentMetadata, str]:
 2        """Extract frontmatter metadata from content.
 3
 4        Parameters
 5        ----------
 6        content : str
 7            Full document content
 8
 9        Returns
10        -------
11        tuple[DocumentMetadata, str]
12            Extracted metadata and remaining content
13
14        """
15        metadata = DocumentMetadata()
16
17        # Option-driven behavior: Respect user preferences
18        if not self.options.include_frontmatter:
19            return metadata, content
20
21        # Format detection: Look for YAML-style frontmatter delimiters
22        # Support both Unix (\n) and Windows (\r\n) line endings
23        if not content.startswith("---\n") and not content.startswith("---\r\n"):
24            return metadata, content
25
26        # Parse frontmatter block: Find closing delimiter
27        lines = content.split("\n")
28        end_index = -1
29        for i in range(1, len(lines)):  # Start at 1 to skip opening "---"
30            if lines[i].strip() == "---":
31                end_index = i
32                break
33
34        # Error handling: Malformed frontmatter
35        if end_index == -1:
36            # No closing delimiter found
37            if self.options.strict_mode:
38                # In strict mode, fail fast with clear error
39                raise ParsingError("Frontmatter block not closed (missing closing '---')")
40            # In lenient mode, warn and continue (treating it as regular content)
41            logger.warning("Frontmatter block not closed, treating as regular content")
42            return metadata, content
43
44        # Parse key-value pairs: Simple "key: value" format
45        frontmatter_lines = lines[1:end_index]
46        for line in frontmatter_lines:
47            line = line.strip()
48            if not line or ":" not in line:
49                continue  # Skip empty lines and malformed entries
50
51            # Use partition to split on first ":" only (allows ":" in values)
52            key, _, value = line.partition(":")
53            key = key.strip()
54            value = value.strip()
55
56            # Map to DocumentMetadata standard fields
57            # This ensures compatibility with all2md's metadata system
58            if key == "title":
59                metadata.title = value
60            elif key == "author":
61                metadata.author = value
62            elif key == "date":
63                # DocumentMetadata has no free-form "date" field (only
64                # creation_date / modification_date), so keep it in custom --
65                # to_dict() flattens custom entries to the top level.
66                metadata.custom["date"] = value
67            elif key == "tags":
68                # Handle comma-separated tags, store as keywords
69                metadata.keywords = [tag.strip() for tag in value.split(",") if tag.strip()]
70            else:
71                # Store custom/unknown fields in metadata.custom dict
72                # This preserves all metadata even if not in standard schema
73                if not metadata.custom:
74                    metadata.custom = {}
75                metadata.custom[key] = value
76

Renderer Implementation

A renderer converts the all2md AST back into your format. The SimpleDoc renderer uses the visitor pattern:

SimpleDocRenderer - Core rendering structure
 1
 2
 3class SimpleDocRenderer(NodeVisitor, InlineContentMixin, BaseRenderer):
 4    """Render AST nodes to SimpleDoc format.
 5
 6    This class implements the visitor pattern to traverse an AST and
 7    generate SimpleDoc output. It demonstrates proper renderer plugin
 8    implementation for the all2md library.
 9
10    SimpleDoc is a minimal format, so some AST node types don't have
11    direct equivalents. This renderer handles them by:
12    - Extracting text content from formatting nodes (strikethrough, underline, etc.)
13    - Skipping unsupported elements (HTML, footnotes, math)
14    - Providing simplified representations where appropriate
15
16    Parameters
17    ----------
18    options : SimpleDocRendererOptions or None
19        Rendering options
20
21    """
22
23    def __init__(self, options: SimpleDocRendererOptions | None = None):
24        """Initialize the SimpleDoc renderer with options."""
25        options = options or SimpleDocRendererOptions()
26        # Multiple inheritance: Initialize all base classes
27        # NodeVisitor provides visit_*() dispatching, InlineContentMixin provides helpers,
28        # BaseRenderer provides common renderer infrastructure
29        BaseRenderer.__init__(self, options)
30        self.options: SimpleDocRendererOptions = options
31        # Output accumulation: Build output incrementally as we traverse the AST
32        # Using a list for efficient string concatenation (vs. repeated string +=)
33        self._output: list[str] = []
34
35    def render_to_string(self, document: Document) -> str:
36        """Render a document AST to SimpleDoc string.
37
38        Parameters
39        ----------
40        document : Document
41            The document node to render
42
43        Returns
44        -------
45        str
46            SimpleDoc format output
47
48        """
49        # Reset output buffer for fresh rendering
50        self._output = []
51        # Visitor pattern: Call accept() on root node to start traversal
52        # This triggers visit_document(), which recursively visits all children
53        document.accept(self)
54        # Finalize: Join accumulated strings and ensure single trailing newline

Visitor Pattern Basics

The renderer implements visit_*() methods for each AST node type:

SimpleDocRenderer.visit_document() - Visitor pattern dispatch
 1        self.write_text_output(text, output)
 2
 3    def visit_document(self, node: Document) -> None:
 4        """Render a Document node.
 5
 6        Parameters
 7        ----------
 8        node : Document
 9            Document to render
10
11        """
12        # Frontmatter rendering: Optional based on settings
13        if self.options.include_frontmatter and node.metadata:
14            self._render_frontmatter(node.metadata)
15
16        # Traverse children: Each child.accept(self) dispatches to the appropriate visit_*() method
17        for i, child in enumerate(node.children):
18            child.accept(self)  # Visitor pattern dispatch
19
20            # Block spacing: Add configurable blank lines between elements
21            # This keeps output readable and matches SimpleDoc conventions

Inline Content Rendering

Use the InlineContentMixin helper for inline nodes:

SimpleDocRenderer.visit_heading() - Inline content extraction
 1        self._output.append("---\n\n")
 2
 3    def visit_heading(self, node: Heading) -> None:
 4        """Render a Heading node.
 5
 6        Parameters
 7        ----------
 8        node : Heading
 9            Heading to render
10
11        """
12        # InlineContentMixin: Use helper to extract text from inline nodes
13        # This handles nested formatting (bold, italic, links) within headings
14        content = self._render_inline_content(node.content)

Renderer Implementation Patterns

When implementing a renderer plugin, you must provide visit_*() methods for all AST node types, even if your format doesn’t support them. There are three standard patterns for handling unsupported nodes:

Pattern 1: Extract Content from Formatting Nodes

For nodes that represent formatting your format doesn’t support (bold, italic, strikethrough, underline, superscript, subscript), extract and render the text content:

Handling unsupported formatting - Extract text content
 1        self._output.append(node.content)
 2
 3    def visit_strong(self, node: Strong) -> None:
 4        """Render a Strong node.
 5
 6        SimpleDoc doesn't have bold formatting, so we just render content.
 7
 8        Parameters
 9        ----------
10        node : Strong
11            Strong to render
12
13        """
14        # Pattern 1: Extract content from unsupported formatting
15        # We preserve the text but lose the bold formatting
16        # This is the standard approach for formatting nodes your format doesn't support

This preserves the text content even if the formatting is lost. Apply this pattern to:

  • Strong (bold)

  • Emphasis (italic)

  • Strikethrough

  • Underline

  • Superscript

  • Subscript

  • Code (inline code)

Pattern 2: Provide Simplified Representation

For elements that have some meaningful equivalent, provide a simplified representation:

Handling links - Include URL in plain text
 1        self._output.append(node.content)
 2
 3    def visit_link(self, node: Link) -> None:
 4        """Render a Link node.
 5
 6        SimpleDoc doesn't have link syntax, so we render the text content
 7        with the URL in parentheses.
 8
 9        Parameters
10        ----------
11        node : Link
12            Link to render
13
14        """
15        # Pattern 2: Provide simplified representation
16        # Instead of just dropping the link, we include the URL in plain text
17        # This preserves information even if not in ideal format
Handling tables - Simplified text representation
 1            child.accept(self)
 2
 3    def visit_table(self, node: Table) -> None:
 4        """Render a Table node.
 5
 6        SimpleDoc doesn't have native table syntax, so we render a
 7        simplified text representation.
 8
 9        Parameters
10        ----------
11        node : Table
12            Table to render
13
14        """
15        # Direct child access pattern: We directly access cell.content instead of calling accept()
16        # This gives us control over formatting while avoiding the need for TableCell visitor logic
17        self._output.append("Table:\n")
18
19        # Render header if present
20        if node.header:
21            self._output.append("Header: ")
22            for i, cell in enumerate(node.header.cells):
23                if i > 0:
24                    self._output.append(" | ")
25                # Extract inline content from cells directly
26                content = self._render_inline_content(cell.content)
27                self._output.append(content)
28            self._output.append("\n")
29
30        # Render data rows
31        for row_idx, row in enumerate(node.rows):
32            self._output.append(f"Row {row_idx + 1}: ")
33            for i, cell in enumerate(row.cells):
34                if i > 0:
35                    self._output.append(" | ")
36                content = self._render_inline_content(cell.content)

This approach preserves information even if not in ideal format. Apply this pattern to:

  • Link - Include URL in parentheses

  • Image - Show alt text or description

  • Table - Render as formatted text

  • DefinitionList - Render as key-value pairs

  • BlockQuote - Render children as-is or with prefix

Pattern 3: Skip Unsupported Elements

For elements your format truly cannot represent, use pass with clear documentation:

Handling HTML blocks - Skip entirely
 1        pass
 2
 3    def visit_html_block(self, node: HTMLBlock) -> None:
 4        """Render an HTMLBlock node.
 5
 6        SimpleDoc doesn't support HTML blocks, so we skip them.
 7        In a real implementation, you might want to extract text content
 8        or provide a placeholder.
 9
10        Parameters
11        ----------
12        node : HTMLBlock
13            HTML block to render
14
15        """
16        # Pattern 3: Skip unsupported elements entirely
17        # Use 'pass' for elements your format truly cannot represent

Always document why you’re skipping an element. Don’t leave empty methods unexplained. Apply this pattern to:

  • HTMLBlock / HTMLInline - Raw HTML

  • FootnoteReference / FootnoteDefinition - Footnotes

  • MathInline / MathBlock - Mathematical notation

  • ThematicBreak - Horizontal rule (if no equivalent)

Pattern 4: Parent-Handled Structural Nodes

Some nodes are only rendered as part of their parent:

Structural nodes handled by parent
 1        self._output.append(content)
 2
 3    def visit_table_cell(self, node: TableCell) -> None:
 4        """Render a TableCell node.
 5
 6        This is handled by visit_table, so this method is a no-op.
 7        The table visitor directly accesses cell content.
 8
 9        Parameters
10        ----------
11        node : TableCell
12            Table cell to render
13
14        """
15        # Structural node pattern: Parent handles rendering
16        # Some nodes (cells, rows, etc.) are only rendered as part of their parent

Apply this pattern to:

  • TableCell - Handled by visit_table

  • TableRow - Handled by visit_table

  • DefinitionTerm / DefinitionDescription - Handled by visit_definition_list

Options Classes

Define custom options for parser and renderer configuration:

SimpleDocOptions - Parser configuration with CLI integration
 1# Frozen dataclass pattern: Make options immutable for thread safety and clarity
 2# frozen=True prevents accidental modification after creation
 3# This is best practice for configuration classes
 4@dataclass(frozen=True)
 5class SimpleDocOptions(BaseParserOptions):
 6    """Options for parsing SimpleDoc documents.
 7
 8    Parameters
 9    ----------
10    include_frontmatter : bool, default=True
11        Whether to parse and include metadata from the frontmatter block.
12        If False, frontmatter is ignored.
13    parse_code_blocks : bool, default=True
14        Whether to parse code blocks (triple backticks).
15        If False, code blocks are treated as regular paragraphs.
16    parse_lists : bool, default=True
17        Whether to parse list items (lines starting with '-').
18        If False, list items are treated as regular paragraphs.
19    strict_mode : bool, default=False
20        If True, raise errors on malformed syntax.
21        If False, attempt to parse gracefully with warnings.
22
23    """
24
25    # Field metadata: Enables CLI integration and documentation
26    # "help": Description shown in --help
27    # "cli_flag": Command-line flag name for this option
28    # The CLI system automatically reads this metadata to generate arguments
29    include_frontmatter: bool = field(
30        default=True,
31        metadata={
32            "help": "Parse and include metadata from frontmatter block",
33            "cli_flag": "--simpledoc-include-frontmatter",
34        },
35    )
36    parse_code_blocks: bool = field(
37        default=True,
38        metadata={
39            "help": "Parse code blocks (triple backticks)",
40            "cli_flag": "--simpledoc-parse-code-blocks",
41        },
42    )
43    parse_lists: bool = field(
44        default=True,
45        metadata={
46            "help": "Parse list items (lines starting with '-')",
47            "cli_flag": "--simpledoc-parse-lists",
48        },
49    )
50    strict_mode: bool = field(
51        default=False,
52        metadata={
53            "help": "Raise errors on malformed syntax instead of parsing gracefully",
54            "cli_flag": "--simpledoc-strict",
55        },
56    )

Key patterns:

  • Frozen dataclass: Use frozen=True for immutability

  • Field metadata: Enables automatic CLI flag generation

  • Validation: Use __post_init__ for validation

__post_init__ validation pattern
 1    def __post_init__(self) -> None:
 2        """Validate option values.
 3
 4        Raises
 5        ------
 6        ValueError
 7            If option values are invalid
 8
 9        """
10        # Always call parent __post_init__ for proper inheritance
11        super().__post_init__()
12
13        # Validate numeric constraints
14        if self.indent_size < 0:
15            raise ValueError(f"indent_size must be non-negative, got {self.indent_size}")
16
17        if self.newlines_between_blocks < 0:
18            raise ValueError(f"newlines_between_blocks must be non-negative, got {self.newlines_between_blocks}")
19
20        # Validate string constraints (markers cannot be empty)
21        if not self.heading_marker:
22            raise ValueError("heading_marker cannot be empty")
23
24        if not self.list_marker:
25            raise ValueError("list_marker cannot be empty")

Metadata Registration

The ConverterMetadata object is the key to plugin discovery:

CONVERTER_METADATA - Plugin registration
 1    # Format identifier: Unique name for your format (lowercase, no spaces)
 2    # Used in CLI (--format simpledoc) and API (get_parser("simpledoc"))
 3    format_name="simpledoc",
 4    # File extensions: List of extensions for auto-detection
 5    # When a file ends with .sdoc, all2md automatically uses this plugin
 6    extensions=[".sdoc", ".simpledoc"],
 7    # MIME types: For web/HTTP-based format detection
 8    # Use standard MIME types if they exist, or invent x- prefixed ones
 9    mime_types=["text/x-simpledoc", "application/x-simpledoc"],
10    # Magic bytes: Binary signatures for format detection from file content
11    # Format: list of (byte_pattern, offset) tuples
12    # This allows detection even when file extension is missing/wrong
13    magic_bytes=[
14        (b"---\n", 0),  # Frontmatter at start of file (Unix line endings)
15        (b"---\r\n", 0),  # Frontmatter with Windows line endings
16    ],
17    # Parser class: Can be direct class reference (shown here) or string path
18    # Direct reference is preferred for plugins (avoids import issues)
19    parser_class=SimpleDocParser,
20    # Renderer class: Enables bidirectional conversion (AST -> SimpleDoc)
21    # If you only need parsing, you can omit this or use a standard renderer
22    renderer_class=SimpleDocRenderer,
23    # Output type: True if renderer produces string, False for binary (bytes)
24    # SimpleDoc is text, so True. PDF/DOCX would be False.
25    renders_as_string=True,
26    # Dependencies: List required packages as (pip_name, import_name, version_spec)
27    # Example: [("python-docx", "docx", ">=0.8.11")]
28    # Empty list means no dependencies
29    parser_required_packages=[],
30    renderer_required_packages=[],
31    # Optional packages: Nice-to-have but not required
32    # Example: [("pillow", "PIL", ">=9.0")] for optional image processing
33    optional_packages=[],
34    # Error message: Shown when required dependencies are missing
35    # Provide helpful installation instructions
36    import_error_message="SimpleDoc conversion requires no external dependencies.",
37    # Options classes: Configuration for parser and renderer
38    # These enable CLI flags and programmatic options
39    parser_options_class=SimpleDocOptions,
40    renderer_options_class=SimpleDocRendererOptions,
41    # Description: Human-readable description for --list-formats
42    description="Convert SimpleDoc lightweight markup format to and from AST. "
43    "SimpleDoc is an educational format demonstrating plugin development.",
44    # Priority: Controls detection order when multiple formats match
45    # Higher numbers checked first. Range: 0-10
46    # 10 = very specific formats (PDF, DOCX)
47    # 5 = medium specificity (our format)
48    # 0 = generic fallbacks (plain text)
49    priority=5,
50)
51

Key fields:

  • format_name: Unique identifier (used in CLI and API)

  • extensions: File extensions for auto-detection

  • mime_types: MIME types for web/HTTP detection

  • magic_bytes: Binary signatures for content-based detection

  • parser_class / renderer_class: Implementation classes

  • parser_options_class / renderer_options_class: Configuration classes

  • parser_required_packages: Dependencies as (pip_name, import_name, version_spec) tuples

  • priority: Detection order (0-10, higher = checked first)

Package Configuration

Register your plugin via entry points in pyproject.toml:

pyproject.toml - Package configuration with entry point
 1[build-system]
 2requires = ["hatchling"]
 3build-backend = "hatchling.build"
 4
 5[project]
 6name = "all2md-simpledoc"
 7version = "1.0.0"
 8description = "SimpleDoc format plugin for all2md - demonstrates bidirectional document conversion"
 9readme = "README.md"
10requires-python = ">=3.10"
11license = {text = "MIT"}
12authors = [
13    {name = "All2md Contributors", email = "example@example.com"}
14]
15keywords = ["all2md", "markdown", "converter", "plugin", "simpledoc"]
16classifiers = [
17    "Development Status :: 4 - Beta",
18    "Intended Audience :: Developers",
19    "License :: OSI Approved :: MIT License",
20    "Programming Language :: Python :: 3",
21    "Programming Language :: Python :: 3.12",
22    "Topic :: Text Processing :: Markup",
23]
24dependencies = [
25    "all2md>=1.0.0",
26]
27
28[project.urls]
29Homepage = "https://github.com/thomas-villani/all2md/tree/main/examples/plugins/simpledoc-plugin"
30Repository = "https://github.com/thomas-villani/all2md.git"
31Issues = "https://github.com/thomas-villani/all2md/issues"
32
33# Entry point for plugin discovery
34[project.entry-points."all2md.converters"]
35simpledoc = "all2md_simpledoc:CONVERTER_METADATA"
36
37[tool.hatch.build.targets.wheel]
38packages = ["src/all2md_simpledoc"]

The critical element is the entry point:

[project.entry-points."all2md.converters"]
simpledoc = "all2md_simpledoc:CONVERTER_METADATA"

This tells all2md to load CONVERTER_METADATA from the all2md_simpledoc package when discovering plugins.

Advanced Features

Custom Options Classes

For complex converters, you may want to define custom options. The parser_options_class and renderer_options_class attributes in ConverterMetadata support two ways to specify options classes:

Method 1: Direct Class Reference (Recommended)

Pass the class object directly:

from all2md.options import BaseParserOptions

CONVERTER_METADATA = ConverterMetadata(
    # ... other fields ...
    parser_options_class=BaseParserOptions,  # Direct class reference
    renderer_options_class=MarkdownRendererOptions,
    # ... rest of metadata ...
)

Method 2: Fully Qualified Class Name (String)

Use a string with the full module path:

CONVERTER_METADATA = ConverterMetadata(
    # ... other fields ...
    parser_options_class="all2md.options.BaseParserOptions",  # String reference
    renderer_options_class="all2md.options.markdown.MarkdownRendererOptions",
    # ... rest of metadata ...
)

Creating Custom Options

For custom options classes in your plugin package:

# all2md_myformat/options.py
from dataclasses import dataclass, field
from all2md.options.base import BaseParserOptions

@dataclass(frozen=True)
class MyFormatOptions(BaseParserOptions):
    """Options for MyFormat conversion."""

    extract_metadata: bool = field(
        default=True,
        metadata={"help": "Extract document metadata"}
    )
    preserve_formatting: bool = field(
        default=False,
        metadata={"help": "Preserve original formatting"}
    )
    custom_parser_mode: str = field(
        default="strict",
        metadata={"help": "Parser mode: strict, lenient, or auto"}
    )

Then reference it in your metadata (either direct or string reference):

from all2md_myformat.options import MyFormatOptions

CONVERTER_METADATA = ConverterMetadata(
    # ... other fields ...
    parser_options_class=MyFormatOptions,  # Direct reference
    # Or: parser_options_class="all2md_myformat.options.MyFormatOptions",
    # ... rest of metadata ...
)

Error Handling

Implement robust error handling in your parser class:

from all2md.exceptions import DependencyError, ParsingError
from all2md.utils.decorators import requires_dependencies

class MyFormatParser(BaseParser):
    """Parser with proper error handling."""

    @requires_dependencies("myformat", [("myformat-lib", "myformat_lib", ">=1.0")])
    def parse(self, input_data):
        """Parse with dependency checking via decorator.

        The @requires_dependencies decorator automatically raises DependencyError
        with correct parameters if the required packages are missing.
        """
        try:
            # Your conversion logic
            pass
        except Exception as e:
            raise ParsingError(
                f"Failed to convert MyFormat document: {e}",
                parsing_stage="parsing",
                original_error=e
            ) from e

    def extract_metadata(self, document):
        """Extract metadata with error handling."""
        try:
            # Metadata extraction logic
            return DocumentMetadata()
        except Exception as e:
            # Log but don't fail - return empty metadata on error
            import logging
            logger = logging.getLogger(__name__)
            logger.warning(f"Failed to extract metadata: {e}")
            return DocumentMetadata()

Manual Dependency Error Handling

If you need to manually raise DependencyError (not recommended - use the decorator instead):

try:
    import myformat_lib
except ImportError as e:
    raise DependencyError(
        converter_name="myformat",
        missing_packages=[("myformat-lib", "myformat_lib", ">=1.0")],
        version_mismatches=None,
        install_command="pip install 'myformat-lib>=1.0'",
        message=None,
        original_import_error=e  # Note: parameter name is original_import_error
    ) from e

Format Detection

The plugin system supports multiple format detection methods:

  1. File extensions: Listed in extensions field

  2. MIME types: Listed in mime_types field

  3. Magic bytes: Binary signatures in magic_bytes field

  4. Priority: Higher priority converters are checked first

Example magic bytes patterns:

magic_bytes=[
    (b"MYFORMAT", 0),      # Exact match at start of file
    (b"<?xml", 0),         # XML declaration
    (b"\x50\x4B", 0),      # ZIP signature (for container formats)
    (b"VERSION", 10),      # Pattern at specific offset
]

Testing Your Plugin

Create comprehensive tests for your plugin. Here’s an example test suite:

# tests/test_myformat_plugin.py
import pytest
from pathlib import Path
from all2md.converter_registry import get_registry
from all2md.ast import Document
from all2md_myformat.parser import CONVERTER_METADATA, MyFormatParser
from all2md.options import BaseParserOptions

def test_plugin_registration():
    """Test that the plugin is properly registered."""
    registry = get_registry()
    assert "myformat" in registry.list_formats()

def test_format_detection():
    """Test format detection by extension and magic bytes."""
    registry = get_registry()
    # Test extension detection
    assert registry.detect_format("test.myf") == "myformat"
    # Test magic bytes detection
    assert registry.detect_format(b"MYFORMAT content here") == "myformat"

def test_parser_instantiation():
    """Test that parser can be instantiated."""
    parser = MyFormatParser()
    assert parser is not None
    assert isinstance(parser.options, BaseParserOptions)

def test_parse_from_bytes():
    """Test parsing from bytes input."""
    parser = MyFormatParser()
    test_content = b"MYFORMAT test document content"
    result = parser.parse(test_content)
    assert isinstance(result, Document)
    assert len(result.children) > 0

def test_parse_from_path(tmp_path):
    """Test parsing from file path."""
    # Create test file
    test_file = tmp_path / "test.myf"
    test_file.write_bytes(b"MYFORMAT test document content")

    # Parse
    parser = MyFormatParser()
    result = parser.parse(test_file)
    assert isinstance(result, Document)
    assert len(result.children) > 0

def test_metadata_extraction():
    """Test metadata extraction."""
    parser = MyFormatParser()
    # Create mock document object
    mock_doc = {"title": "Test Document", "author": "Test Author"}
    metadata = parser.extract_metadata(mock_doc)
    assert metadata is not None
    # Add assertions based on your metadata extraction logic

def test_error_handling():
    """Test that appropriate errors are raised."""
    parser = MyFormatParser()
    with pytest.raises(Exception):  # ParsingError or similar
        parser.parse(b"INVALID content that should fail")

def test_with_options():
    """Test parsing with custom options."""
    options = BaseParserOptions()
    parser = MyFormatParser(options=options)
    result = parser.parse(b"MYFORMAT test content")
    assert isinstance(result, Document)

Testing Renderer

Test the renderer with a variety of AST structures:

def test_renderer_basic():
    """Test basic rendering."""
    from all2md.ast import Document, Paragraph, Text
    from all2md_myformat.renderer import MyFormatRenderer

    doc = Document(children=[
        Paragraph(content=[Text(content="Hello, world!")])
    ])

    renderer = MyFormatRenderer()
    output = renderer.render_to_string(doc)
    assert "Hello, world!" in output

def test_unsupported_formatting_preserves_content():
    """Test that unsupported formatting still renders text content."""
    from all2md.ast import Document, Paragraph, Strikethrough, Text
    from all2md_myformat.renderer import MyFormatRenderer

    doc = Document(children=[
        Paragraph(content=[
            Strikethrough(content=[Text(content="crossed out")])
        ])
    ])

    renderer = MyFormatRenderer()
    output = renderer.render_to_string(doc)

    # Content should be preserved even without strikethrough
    assert "crossed out" in output

def test_complex_document():
    """Test rendering a complex document with all node types."""
    from all2md.ast import (
        Document, Heading, Paragraph, List, ListItem,
        CodeBlock, Strong, Emphasis, Link, Image, Text
    )
    from all2md_myformat.renderer import MyFormatRenderer

    # Build complex document with many node types
    doc = Document(children=[
        Heading(level=1, content=[Text(content="Title")]),
        Paragraph(content=[
            Strong(content=[Text(content="bold")]),
            Text(content=" and "),
            Emphasis(content=[Text(content="italic")]),
        ]),
        List(ordered=False, items=[
            ListItem(children=[Paragraph(content=[Text(content="Item 1")])]),
            ListItem(children=[Paragraph(content=[Text(content="Item 2")])]),
        ]),
        CodeBlock(language="python", content="print('hello')"),
    ])

    renderer = MyFormatRenderer()
    output = renderer.render_to_string(doc)

    # Verify all content is present
    assert "Title" in output
    assert "bold" in output
    assert "italic" in output
    assert "Item 1" in output
    assert "print('hello')" in output

Publishing Your Plugin

Naming Convention

Follow the naming convention all2md-{format} for your package name to make it easily discoverable.

Distribution

  1. Build your package:

    python -m build

  2. Upload to PyPI:

    python -m twine upload dist/*

  3. Installation by users:

    pip install all2md-myformat

The plugin will be automatically discovered when all2md is imported.

Best Practices

  1. Comprehensive format detection: Implement robust magic byte detection for reliable format identification

  2. Graceful degradation: Handle missing optional dependencies gracefully

  3. Clear error messages: Provide helpful error messages with installation instructions

  4. Documentation: Include clear documentation and usage examples

  5. Testing: Test with various file types and edge cases

  6. Performance: Optimize for large files and consider memory usage

  7. Compatibility: Ensure compatibility with all supported Python versions

  8. Complete visitor implementation: Implement all visit_*() methods in renderers, even if just pass

  9. Document unsupported features: Clearly document which AST node types your format doesn’t support

Example Plugins

Here are some ideas for useful plugins:

  • all2md-visio: Microsoft Visio diagrams

  • all2md-dwg: AutoCAD drawings

  • all2md-pages: Apple Pages documents

  • all2md-confluence: Confluence wiki pages

  • all2md-notion: Notion exports

  • all2md-custom-transforms: Custom AST transforms for specialized workflows

Reference Implementation

The complete SimpleDoc plugin serves as a reference implementation:

  • Source code: examples/plugins/simpledoc-plugin/

  • Parser: src/all2md_simpledoc/parser.py (full bidirectional parser)

  • Renderer: src/all2md_simpledoc/renderer.py (complete visitor implementation)

  • Options: src/all2md_simpledoc/options.py (parser and renderer options)

  • Metadata: src/all2md_simpledoc/__init__.py (plugin registration)

  • Tests: tests/test_simpledoc.py (comprehensive test coverage)

  • README: Detailed usage examples and format specification

Community

Support

If you encounter issues developing plugins:

  1. Check the SimpleDoc plugin example in examples/plugins/simpledoc-plugin/

  2. Review the RENDERER_PATTERNS.md guide in the simpledoc-plugin directory

  3. Review existing plugin implementations

  4. Open an issue with the plugin-development label