all2md.ast.splitting

Document splitting strategies.

This module provides strategies for splitting documents at semantic boundaries based on different criteria: heading levels, word counts, number of parts, thematic breaks, or automatic detection.

Classes

SplitResult : Represents a split portion with metadata DocumentSplitter : Main class for document splitting strategies

Functions

parse_split_spec : Parse –split-by CLI argument

Examples

Split by heading level:
>>> splitter = DocumentSplitter()
>>> splits = splitter.split_by_heading_level(doc, level=1)
>>> for split in splits:
...     print(f"Part {split.index}: {split.title} ({split.word_count} words)")
Split by word count:
>>> splits = splitter.split_by_word_count(doc, target_words=500)
Split into equal parts:
>>> splits = splitter.split_by_parts(doc, num_parts=5)
Auto-detect best strategy:
>>> splits = splitter.split_auto(doc)
Split by sections:
>>> splits = splitter.split_by_sections(doc, include_preamble=True)
class all2md.ast.splitting.SplitResult

Bases: object

Represents a split portion of a document.

Variables:

  • document (Document) – The split document AST

  • index (int) – 1-based index of this split (001, 002, etc.)

  • title (Optional[str]) – Title/heading text for this split (if available)

  • word_count (int) – Approximate word count for this split

  • metadata (dict) – Additional metadata for this split

document: Document
index: int
title: str | None = None
word_count: int = 0
metadata: dict
get_filename_slug() → str

Generate filesystem-safe slug from title.

Returns:

Sanitized slug suitable for filenames

Return type:

str

Examples

>>> split = SplitResult(doc, 1, title="Chapter 1: Introduction")
>>> split.get_filename_slug()
'chapter-1-introduction'
__init__(document: Document, index: int, title: str | None = None, word_count: int = 0, metadata: dict = <factory>) → None
class all2md.ast.splitting.DocumentSplitter

Bases: object

Handles various document splitting strategies.

This class provides methods to split documents at semantic boundaries based on different criteria: heading levels, word counts, number of parts, or automatic detection.

static split_by_heading_level(doc: Document, level: int, include_preamble: bool = True) → list[SplitResult]

Split document at every heading of specified level.

Parameters:

  • doc (Document) – Document to split

  • level (int) – Heading level to split on (1-6)

  • include_preamble (bool) – Whether to include content before first heading as separate split

Returns:

Split documents, one per section at specified level

Return type:

list of SplitResult

Raises:

ValueError – If level is not between 1 and 6

Examples

>>> DocumentSplitter.split_by_heading_level(doc, level=1)
static split_by_word_count(doc: Document, target_words: int) → list[SplitResult]

Split document by word count, maintaining section boundaries.

Accumulates sections until target word count is reached, then creates a split. Ensures splits occur at section boundaries for semantic coherence.

Parameters:

  • doc (Document) – Document to split

  • target_words (int) – Target word count per split (approximate)

Returns:

Split documents with roughly equal word counts

Return type:

list of SplitResult

Raises:

ValueError – If target_words is less than 1

Examples

>>> DocumentSplitter.split_by_word_count(doc, target_words=500)
static split_by_parts(doc: Document, num_parts: int) → list[SplitResult]

Split document into N roughly equal parts at section boundaries.

Calculates total word count and divides by num_parts to determine target words per part. Then uses word count splitting to create approximately equal splits.

Parameters:

  • doc (Document) – Document to split

  • num_parts (int) – Number of parts to create

Returns:

Split documents with roughly equal word counts

Return type:

list of SplitResult

Raises:

ValueError – If num_parts is less than 1

Examples

>>> DocumentSplitter.split_by_parts(doc, num_parts=5)
static split_into_slices(doc: Document, num_slices: int) → list[SplitResult]

Divide a document into exactly num_slices balanced, contiguous slices.

Unlike split_by_parts() (which targets a word count and may yield a different number of parts), this guarantees exactly num_slices slices whenever the document has at least that many semantic “atoms” (sections, or word-count blocks for heading-light documents). When the document has fewer atoms than requested, it returns one slice per atom.

This powers the --slice X/Y paging flag, where a deterministic Y matters so callers can page 1/Y, 2/Y, ... Y/Y.

Parameters:

  • doc (Document) – Document to slice.

  • num_slices (int) – Desired number of slices (Y). Must be >= 1.

Returns:

min(num_slices, num_atoms) contiguous slices in document order.

Return type:

list of SplitResult

Raises:

ValueError – If num_slices is less than 1.

static split_by_break(doc: Document) → list[SplitResult]

Split document at thematic breaks (horizontal rules).

Splits the document at any ThematicBreak nodes, which represent horizontal rules (---, ***, ___) in Markdown and similar separators in other formats.

Parameters:

doc (Document) – Document to split

Returns:

Split documents at thematic break boundaries

Return type:

list of SplitResult

Examples

>>> DocumentSplitter.split_by_break(doc)
static split_by_delimiter(doc: Document, delimiter: str) → list[SplitResult]

Split document at custom text delimiters.

Searches for paragraphs or text nodes that contain only the delimiter text (allowing for whitespace) and splits the document at those points.

Parameters:

  • doc (Document) – Document to split

  • delimiter (str) – Text delimiter to split on (e.g., "-----", "***", "<!-- split -->")

Returns:

Split documents at delimiter boundaries

Return type:

list of SplitResult

Examples

>>> DocumentSplitter.split_by_delimiter(doc, delimiter="-----")
static split_auto(doc: Document, target_words: int = 1500) → list[SplitResult]

Automatically determine best split strategy based on document structure.

Analyzes document to find natural split points: 1. Try h1 boundaries if sections are reasonable size 2. Otherwise try h2 boundaries 3. Fall back to word count splitting if sections too large

Parameters:

  • doc (Document) – Document to split

  • target_words (int) – Target word count per split for fallback strategy

Returns:

Split documents using the best detected strategy

Return type:

list of SplitResult

Examples

>>> DocumentSplitter.split_auto(doc)  # Target ~1500 words per split
static split_by_sections(doc: Document, include_preamble: bool = True) → list[SplitResult]

Split document into separate documents by sections.

This method was moved from document_utils.py and adapted to return list[SplitResult] for consistency with other DocumentSplitter methods.

Parameters:

  • doc (Document) – Document to split

  • include_preamble (bool, default = True) – If True and there is content before the first heading, include it as a separate split at the beginning

Returns:

List of split results, one per section (plus preamble if present)

Return type:

list of SplitResult

Examples

>>> splits = DocumentSplitter.split_by_sections(doc)
>>> for i, split_result in enumerate(splits):
...     print(f"Section {i}: {len(split_result.document.children)} nodes")
all2md.ast.splitting.parse_split_spec(spec: str) → tuple[str, Any]

Parse –split-by CLI argument into strategy and parameters.

Parameters:

spec (str) – Split specification string

Returns:

(strategy_name, parameter) where: - (“heading”, 1) for “h1” - (“heading”, 2) for “h2” - (“length”, 400) for “length=400” - (“parts”, 4) for “parts=4” - (“delimiter”, “—–”) for “delimiter=—–” - (“break”, None) for “break” - (“page”, None) for “page” - (“chapter”, None) for “chapter” - (“auto”, None) for “auto”

Return type:

tuple of (str, Any)

Raises:

ValueError – If spec format is invalid

Examples

>>> parse_split_spec("h1")
('heading', 1)
>>> parse_split_spec("length=500")
('length', 500)
>>> parse_split_spec("parts=3")
('parts', 3)
>>> parse_split_spec("delimiter=-----")
('delimiter', '-----')
>>> parse_split_spec("auto")
('auto', None)