all2md.utils.encoding

Character encoding detection and handling utilities.

This module provides utilities for detecting and handling character encodings in text-based files, with support for chardet-based detection and fallback strategies for maximum compatibility.

all2md.utils.encoding.detect_encoding(data: bytes, sample_size: int = 8192, confidence_threshold: float = 0.7) → str | None

Detect character encoding of binary data using chardet.

Parameters:

  • data (bytes) – Binary data to analyze

  • sample_size (int, default 8192) – Number of bytes to sample for detection (uses first N bytes)

  • confidence_threshold (float, default 0.7) – Minimum confidence level (0.0-1.0) required to trust detection

Returns:

Detected encoding name (e.g., ‘utf-8’, ‘latin-1’), or None if: - chardet is not available - detection fails - confidence is below threshold

Return type:

str | None

Examples

>>> data = b"Hello, world!"
>>> encoding = detect_encoding(data)
>>> if encoding:
...     text = data.decode(encoding)
all2md.utils.encoding.read_text_with_encoding_detection(data: bytes, fallback_encodings: list[str] | None = None, use_chardet: bool = True, chardet_sample_size: int = 8192, chardet_confidence_threshold: float = 0.7) → str

Read binary data as text with automatic encoding detection.

Attempts to decode binary data using multiple strategies: 1. chardet-based detection (if enabled and available) 2. Fallback encodings in order 3. Final fallback with error replacement

Parameters:

  • data (bytes) – Binary data to decode

  • fallback_encodings (list[str] | None, default None) – List of encodings to try in order. If None, uses: [‘utf-8’, ‘utf-8-sig’, ‘latin-1’]

  • use_chardet (bool, default True) – Whether to attempt chardet-based detection first

  • chardet_sample_size (int, default 8192) – Number of bytes to sample for chardet detection

  • chardet_confidence_threshold (float, default 0.7) – Minimum confidence for chardet detection

Returns:

Decoded text content

Return type:

str

Raises:

ValueError – If data cannot be decoded with any encoding (should be rare since latin-1 accepts any byte sequence)

Examples

>>> data = b"Hello, world!"
>>> text = read_text_with_encoding_detection(data)
>>> print(text)
Hello, world!

>>> # Custom fallback encodings
>>> text = read_text_with_encoding_detection(
...     data,
...     fallback_encodings=['cp1252', 'utf-8', 'latin-1']
... )
all2md.utils.encoding.get_charset_from_content_type(content_type: str) → str | None

Extract charset parameter from Content-Type header.

Parameters:

content_type (str) – Content-Type header value (e.g., ‘text/html; charset=utf-8’)

Returns:

Charset value if present, None otherwise

Return type:

str | None

Examples

>>> get_charset_from_content_type('text/html; charset=utf-8')
'utf-8'
>>> get_charset_from_content_type('text/html')
None
all2md.utils.encoding.normalize_stream_to_text(stream: IO[bytes] | IO[str], fallback_encodings: list[str] | None = None, use_chardet: bool = True, chardet_sample_size: int = 8192, chardet_confidence_threshold: float = 0.7) → str

Read content from a file-like object and normalize to text.

This helper handles both binary-mode streams (IO[bytes]) and text-mode streams (IO[str]): - Binary streams: decodes using automatic encoding detection - Text streams: returns content as-is

This is useful for parsers that need to accept both binary and text mode file-like objects without crashing on type mismatches.

Parameters:

  • stream (IO[bytes] or IO[str]) – File-like object to read from. Can be either binary mode (e.g., io.BytesIO, open(file, ‘rb’)) or text mode (e.g., io.StringIO, open(file, ‘r’))

  • fallback_encodings (list[str] or None, optional) – List of encodings to try if chardet detection fails. If None, uses [‘utf-8’, ‘utf-8-sig’, ‘latin-1’]

  • use_chardet (bool, default True) – Whether to use chardet for automatic encoding detection on binary streams

  • chardet_sample_size (int, default 8192) – Number of bytes to sample for chardet detection

  • chardet_confidence_threshold (float, default 0.7) – Minimum confidence level for chardet detection

Returns:

Decoded text content from the stream

Return type:

str

Raises:

TypeError – If stream.read() returns something other than bytes or str

Examples

>>> from io import BytesIO, StringIO
>>> # Binary stream
>>> binary_stream = BytesIO(b"Hello, world!")
>>> text = normalize_stream_to_text(binary_stream)
>>> print(text)
Hello, world!

>>> # Text stream
>>> text_stream = StringIO("Hello, world!")
>>> text = normalize_stream_to_text(text_stream)
>>> print(text)
Hello, world!
all2md.utils.encoding.normalize_stream_to_bytes(stream: IO[bytes] | IO[str], encoding: str = 'utf-8') → bytes

Read content from a file-like object and normalize to bytes.

This helper handles both binary-mode streams (IO[bytes]) and text-mode streams (IO[str]): - Binary streams: returns content as-is - Text streams: encodes using specified encoding

This is useful for parsers that need to work with binary data (e.g., for format detection, ZIP signatures, JSON decoding) but may receive either binary or text mode streams.

Parameters:

  • stream (IO[bytes] or IO[str]) – File-like object to read from. Can be either binary mode (e.g., io.BytesIO, open(file, ‘rb’)) or text mode (e.g., io.StringIO, open(file, ‘r’))

  • encoding (str, default "utf-8") – Encoding to use when converting text streams to bytes

Returns:

Content as bytes

Return type:

bytes

Raises:

TypeError – If stream.read() returns something other than bytes or str

Examples

>>> from io import BytesIO, StringIO
>>> # Binary stream
>>> binary_stream = BytesIO(b"Hello, world!")
>>> data = normalize_stream_to_bytes(binary_stream)
>>> print(data)
b'Hello, world!'

>>> # Text stream
>>> text_stream = StringIO("Hello, world!")
>>> data = normalize_stream_to_bytes(text_stream)
>>> print(data)
b'Hello, world!'