all2md.utils.text

Text processing utilities for AST transforms.

This module provides text manipulation functions commonly used by transforms, including slugification for heading IDs and table of contents generation.

Functions

slugify : Convert text to URL-safe slug make_unique_slug : Generate unique slug with duplicate handling

Examples

Basic slugification:

>>> from all2md.utils.text import slugify
>>> slugify("My Heading Title")
'my-heading-title'

Custom separator:

>>> slugify("My Heading Title", separator="_")
'my_heading_title'

Unique slug generation with counter:

>>> seen = {}
>>> slug1 = make_unique_slug("my-heading", seen)  # 'my-heading'
>>> slug2 = make_unique_slug("my-heading", seen)  # 'my-heading-2'
>>> slug3 = make_unique_slug("my-heading", seen)  # 'my-heading-3'

all2md.utils.text.slugify(text: str, *, seen_slugs: Set[str] | None = None, max_length: int = 100, separator: str = '-') → str

Create a URL-safe slug from text with collision avoidance.

This function generates GitHub-flavored Markdown compatible slugs by: - Normalizing Unicode characters (NFD decomposition) - Converting to lowercase - Replacing spaces and underscores with hyphens - Removing non-alphanumeric characters (except hyphens) - Collapsing multiple consecutive hyphens - Stripping leading/trailing hyphens - Handling collisions by appending -2, -3, etc. - Limiting length to max_length characters

Parameters:

• text (str) – Text to slugify (e.g., heading text, filename)

• seen_slugs (Set[str] or None, default = None) – Set of previously generated slugs for collision detection. If provided and the generated slug already exists, a numeric suffix will be appended (-2, -3, etc.). The function will automatically add the new slug to this set.

• max_length (int, default = 100) – Maximum length of the slug. Slugs longer than this will be truncated before adding collision suffixes.

• separator (str, default = "-") – The separator between words in the slug

Returns:

URL-safe slug, unique if seen_slugs is provided

Return type:

str

Examples

Basic slugification:

>>> slugify("Hello World!")
'hello-world'

Handle special characters:

>>> slugify("API Reference (v2.0)")
'api-reference-v20'

Collision detection:

>>> seen = set()
>>> slug1 = slugify("Introduction", seen_slugs=seen)
>>> slug1
'introduction'
>>> slug2 = slugify("Introduction", seen_slugs=seen)
>>> slug2
'introduction-2'

Length limiting:

>>> slugify("A" * 150, max_length=50)
'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'

Unicode normalization:

>>> slugify("Café résumé")
'cafe-resume'

all2md.utils.text.make_unique_slug(slug: str, seen_slugs: dict[str, int], separator: str = '-') → str

Generate unique slug with duplicate handling.

This function ensures slug uniqueness by appending a numeric suffix when duplicates are encountered. The seen_slugs dictionary tracks occurrence counts and is mutated in-place.

Parameters:

• slug (str) – Base slug to make unique

• seen_slugs (dict[str, int]) – Dictionary tracking occurrence counts (mutated in-place). Maps base slug to count of occurrences.

• separator (str, default = "-") – Separator to use before numeric suffix

Returns:

Unique slug (with numeric suffix if needed)

Return type:

str

Examples

Basic usage:

>>> seen = {}
>>> make_unique_slug("my-heading", seen)
'my-heading'
>>> make_unique_slug("my-heading", seen)
'my-heading-2'
>>> make_unique_slug("my-heading", seen)
'my-heading-3'

Custom separator:

>>> seen = {}
>>> make_unique_slug("heading", seen, separator="_")
'heading'
>>> make_unique_slug("heading", seen, separator="_")
'heading_2'

Notes

The seen_slugs dictionary is modified in-place to track counts. This allows callers to maintain state across multiple calls.

The first occurrence of a slug does not get a numeric suffix. Subsequent occurrences get suffixes starting at 2.