all2md.optimize

Auto-tune converter options against a reference-free fidelity objective.

Difficult documents — the scanned PDF, the three-column report, the table with no ruling lines — rarely come with a reference to diff against, so the settings that convert them well cannot be found by comparing to a known-good output. This module searches the option space directly and ranks candidates by how much well-formed structure each one recovers.

Why not reuse an existing score

Neither shipped score works as a search objective:

all2md.confidence is a saturating breakage detector (100 - text_density - ocr_reliance - degraded_events). It answers “can I trust this conversion?”, and on any document that is not visibly broken it pins to 100 regardless of the settings used. Measured across 16 option combinations on a two-column fixture it produced one distinct score while the parsed AST produced four distinct outcomes: no gradient to climb.
all2md.roundtrip needs a ground truth. It manufactures one by re-parsing rendered output, which measures the renderer, not the parser — a garbled table round-trips through Markdown perfectly. It is the right objective for tuning renderer options and the wrong one for tuning a PDF parser.

So the objective here is computed from the parsed AST, which is where the gradient actually lives. The confidence report is still used, but for its penalties — the degraded-content incidents it records are a genuine signal that a conversion broke.

The over-detection trap

“More structure is better” is the obvious objective and it is wrong twice over.

It rewards keeping the boilerplate. Leaving a running header and page footer in yields strictly more text than trimming them, so a naive text-volume objective prefers the worse conversion. Measured against a fixture whose correct parse is known, that objective was anti-correlated with the truth (Pearson r = -0.88): it picked the worst candidate nearly every time. Text is therefore scored over body content only, with repeated furniture excluded.

Getting that exclusion right took two corrections, both forced by measurement:

Block-level deduplication is not enough. The parser frequently glues a footer into an adjacent body block (“Page 1 of 2 | ACME CONFIDENTIAL Beta sentence B1 …”). That block is unique, so no block comparison can ever see it. Furniture must be found as a repeated word sequence spanning blocks — find_furniture().
Furniture is a property of the document, not of one parse of it. Deriving it per-candidate lets a candidate whose block segmentation happens to hide the repetition escape detection and bank its running header as recovered body text. The candidates then stop tying on text and keeping the boilerplate wins again. score_candidates() therefore pools what every candidate revealed and applies the union to all of them. This one bit differs between platforms and PyMuPDF versions, so a per-candidate rule can look perfectly correct locally and invert the ranking somewhere else.

With furniture excluded, a trimmed and an untrimmed conversion tie on text, which frees the cleanliness dimension to break the tie in favour of the one that actually removed it. On the same fixture that took the objective from r = -0.88 to r = +1.00.

It rewards an over-eager table detector, because any junk region promoted to a table adds cells. But scoring well-formedness alone — the obvious correction — simply inverts the bias into rewarding an under-eager one: finding five clean tables and missing the sixth then beats finding all six with one messy. That is not theoretical; on a real arXiv paper table_detection_mode="ruling" scored 0.98 on well-formedness against "pymupdf"’s 0.68 while recovering fewer tables. Tables are therefore scored on quality-weighted recall (_table_shape()): filled cells, discounted by shape regularity. A hallucinated table is sparse and ragged and contributes almost nothing; a missed real one costs its cells.

And furniture must repeat like furniture. A sequence counted as boilerplate if it appeared in merely two blocks — correct only for the two-page fixture it was written against. On a 21-page paper it fired on ordinary recurring academic phrasing and flagged 746 words of real body text as boilerplate (against 131 with the corrected rule). False furniture is not harmless: it makes body text look like clutter, so a setting that deletes that text scores as if it had tidied up. The bar now scales with page count — furniture_threshold().

Body text is not a dimension at all

It gates the others. Losing a paragraph is data loss; leaving a running header in is an annoyance, and no exchange rate between them is defensible. As a weighted dimension the two were interchangeable at 0.45 versus 0.15, which means a setting that destroyed 1% of the body could buy its way back with a 3% tidiness gain. That is the wrong trade at any exchange rate, so retention instead multiplies the score, raised to TEXT_RETENTION_EXPONENT: shedding 1% of the body costs ~3% of fitness and no amount of polish buys it back. This is a deliberately conservative posture — the optimizer’s advice is only worth taking if it cannot destroy content.

Fitness is relative to the candidate pool, not absolute: the best text recovery observed across the candidates stands in for the reference we do not have. This is deliberate. The job is to rank candidates, and inventing an absolute 0-100 quality number would both duplicate all2md.confidence and imply a precision these signals do not support.

Every correction above was forced by measurement, and several of them by documents from the wild, which broke an objective that looked perfectly healthy on a synthetic fixture. Changing the scoring without re-running that evaluation is not advisable.

all2md.optimize.BOILERPLATE_NGRAM = 5: Length of the word window used to spot repeated furniture. Long enough that ordinary prose does not collide by chance, short enough to catch a page footer.

all2md.optimize.DIMENSION_WEIGHTS: dict[str, float] = {'cleanliness': 0.3, 'structure': 0.25, 'tables': 0.45}

How the fitness dimensions are combined. Dimensions the candidate pool does not exercise (no candidate found a table) are dropped and the rest renormalized, so a table-free document is neither rewarded nor punished for the tables it lacks.

Body text is deliberately not here. It is not a dimension to be traded against the others; it is a multiplicative gate – see TEXT_RETENTION_EXPONENT.

all2md.optimize.FORBIDDEN_KNOBS: dict[str, str] = {'consolidate_inline_formatting': "correctness, not a trade-off: disabling it fragments inline runs ('hello' -> 'hel' + 'lo')", 'include_comments': 'a content-inclusion preference: more words always score, so this would always be recommended', 'merge_hyphenated_words': 'correctness, not a trade-off: the repair joins two tokens into one, so disabling it games a word count'}

Settings that must never be searched, and why. Enforced by a test, because every entry here was added after the optimizer found and exploited it.

A knob only belongs in KNOBS if it is a genuine fidelity trade-off – a setting where the better value depends on the document and cannot be known in advance. Two other kinds of setting look superficially tunable and are not:

Correctness settings, where one value is simply right. merge_hyphenated_words repairs a word broken across a line break; leaving it off is a defect, not a trade-off. The optimizer recommended disabling it on 17 of 17 real papers – and ground truth confirmed it made every one of them worse – because the repair joins two tokens into one and so looks like losing a word. (content_tokens() now removes that incentive, but the setting still has no business being searched: there is no document for which the broken word is the better answer.)

Content-inclusion preferences, which change what the user asked for rather than how well it was extracted. The objective rewards recovering more content, so it will always say yes to include_comments – not because leaking reviewer comments into the output improves fidelity, but because comments are words and words score. That is a tautology dressed up as a finding, and for comments specifically it is advice that leaks content the author never meant to publish.

all2md.optimize.MINIMIZE_TOLERANCE = 1e-09: How much fitness a recommended setting must be worth to survive the minimization pass. Anything that can be removed without dropping below this is a passenger the search happened to pick up, not a finding, and reporting it as advice is worse than saying nothing: the user cannot tell the difference.

all2md.optimize.MIN_TABLE_CELLS = 8.0

How much of a dimension the best candidate must actually find before that dimension is allowed to influence the ranking.

Every dimension is normalized against the best candidate in the pool, which turns a trivial absolute difference into a total relative one. On a real arXiv paper with no real tables at all, one setting happened to find a single two-column table worth four cells while the default found none: the tables dimension read 1.0 against 0.0 – a maximal swing, worth its full weight – and the optimizer reported a 45 point gain for a change that measurably did nothing. A dimension the document barely exercises carries no information about it, and must not be allowed to decide the ranking. Below these floors it is dropped, exactly as it already is at zero.

all2md.optimize.TEXT_RETENTION_EXPONENT = 3.0

How steeply losing body text is punished. Retention (this candidate’s body words over the best candidate’s) multiplies the whole score raised to this power, so shedding 1% of the body costs roughly 3% of fitness.

This exists because losing a paragraph and keeping a running header are not commensurable defects. As a mere weighted dimension, body text was interchangeable with tidiness, so a setting that destroyed content could buy its way back by removing clutter. A conservative posture is the point: advice that can silently delete text is not advice worth taking.

all2md.optimize.KNOBS: dict[str, dict[str, list[Any]]] = {'docx': {'include_endnotes': [True, False], 'include_footnotes': [True, False], 'include_image_captions': [True, False], 'preserve_tables': [True, False]}, 'html': {'collapse_whitespace': [True, False], 'details_parsing': ['blockquote', 'html', 'skip'], 'detect_table_alignment': [True, False], 'extract_microdata': [True, False], 'extract_readable': [True, False], 'figures_parsing': ['image_with_caption', 'caption_only', 'paragraph', 'skip']}, 'pdf': {'auto_trim_headers_footers': [True, False], 'column_detection_mode': ['auto', 'force_single', 'force_multi'], 'dedup_running_headings': [True, False], 'detect_columns': [True, False], 'detect_merged_cells': [True, False], 'enable_table_fallback_detection': [True, False], 'table_detection_mode': ['pymupdf', 'ruling', 'both', 'none'], 'table_fallback_extraction_mode': ['none', 'grid', 'text_clustering'], 'trim_headers_footers': [True, False]}}: The option values worth searching, per format. Deliberately curated rather than derived from the dataclass fields: most options are irrelevant to fidelity (password), or are a security posture that an optimizer has no business flipping (strip_dangerous_elements), or would explode the search space for no gain. Only knobs that are a genuine fidelity trade-off belong here – see FORBIDDEN_KNOBS for the two categories that are not.

class all2md.optimize.Candidate

Bases: object

One point in the option space, with its measured yield and fitness.

options: dict[str, Any]: The options that differ from the parser’s defaults, e.g. {"detect_columns": True}.

origin: str = 'default'

"default", "preset:quality", "refine:detect_columns".

Type:: Where the candidate came from

metrics: DocumentMetrics

fitness: float = 0.0: Pool-relative fitness, 0-100. Only comparable within one run.

dimensions: dict[str, float]: Per-dimension contributions, for explaining why a candidate won.

to_dict() → dict[str, Any]: Return a JSON-serializable view.

__init__(options: dict[str, ~typing.Any]=<factory>, origin: str = 'default', metrics: DocumentMetrics = <factory>, fitness: float = 0.0, dimensions: dict[str, float]=<factory>) → None

class all2md.optimize.DocumentMetrics

Bases: object

The reference-free structural yield of one conversion.

Everything here is read off the parsed AST, except breakage, which comes from the confidence report’s degraded-content incidents.

blocks: int = 0

words: int = 0: Words in the document, counting every block.

boilerplate_words: int = 0: Words belonging to repeated furniture (running headers, page footers), whether they sit in their own block or were glued into a body block.

unique_words: int = 0

the body content actually recovered. This, not words, is what the text dimension scores.

Type:: words minus the furniture

duplicate_blocks: int = 0: Blocks whose text is entirely a repeat of another block.

headings: int = 0

list_items: int = 0

links: int = 0

tables: int = 0

table_cells: int = 0

good_cells: float = 0.0: Filled cells discounted by shape regularity, summed over tables with 2+ columns. Quality-weighted recall: this, not table_quality, is what the objective scores. Well-formedness alone rewards missing real tables; count alone rewards inventing them.

table_fill: float = 0.0: Fraction of table cells that are non-empty. A hallucinated table is sparse.

table_regularity: float = 0.0: Fraction of table rows whose column count matches the table’s modal count. A hallucinated table is ragged.

breakage: float = 0.0

how much real breakage the converter reported.

Type:: 100 - confidence.score

block_texts: list[str]: The document’s top-level block texts. Kept so the furniture found by any candidate can be re-applied to this one – see score_candidates().

furniture: Furniture: The repeated content this parse revealed.

min_furniture_blocks: int = 2: How many distinct blocks a sequence must span before it counts as furniture. Scales with page count: furniture repeats page after page, ordinary prose does not.

property table_quality: float: Well-formedness of the tables found, 0.0-1.0. Zero if there are none.

to_dict() → dict[str, Any]: Return a JSON-serializable view, including the derived table quality.

__init__(blocks: int = 0, words: int = 0, boilerplate_words: int = 0, unique_words: int = 0, duplicate_blocks: int = 0, headings: int = 0, list_items: int = 0, links: int = 0, tables: int = 0, table_cells: int = 0, good_cells: float = 0.0, table_fill: float = 0.0, table_regularity: float = 0.0, breakage: float = 0.0, block_texts: list[str] = <factory>, furniture: Furniture = <factory>, min_furniture_blocks: int = 2) → None

class all2md.optimize.Furniture

Bases: NamedTuple

The repeated content of a document: running headers, footers, watermarks.

Two shapes, because one alone is not enough:

sequences – word windows appearing in two or more distinct blocks. Needed because a header or footer is frequently glued into an adjacent body block by the parser (“Page 1 of 2 | ACME CONFIDENTIAL Beta sentence B1 …”). Such a block is unique – it contains body text – so no block-level comparison can ever flag it, and its footer words would count as recovered body content.
blocks – whole blocks that repeat. Needed because a short running heading (“Quarterly Report”) is below the n-gram window and no sequence can catch it.

Create new instance of Furniture(sequences, blocks)

sequences: set[tuple[str, ...]]: Alias for field number 0

blocks: set[str]: Alias for field number 1

union(other: Furniture) → Furniture: Combine what two parses each revealed about the document’s furniture.

static __new__(_cls, sequences: set[tuple[str, ...]], blocks: set[str]): Create new instance of Furniture(sequences, blocks)

class all2md.optimize.OptimizationReport

Bases: object

The outcome of a search: what won, what it beat, and by how much.

source_format: str = ''

best_options: dict[str, Any]: The winning options, as a flat {option: value} diff from the defaults.

best_fitness: float = 0.0

baseline_fitness: float = 0.0: Fitness of the parser’s stock defaults, so the gain is legible.

candidates: list[Candidate]: Every candidate evaluated, best first.

evaluated: int = 0

property gain: float: How much fitness the winning options add over the stock defaults.

property improved: bool: Whether the search beat the defaults at all.

to_dict() → dict[str, Any]: Return a JSON-serializable view.

__init__(source_format: str = '', best_options: dict[str, ~typing.Any]=<factory>, best_fitness: float = 0.0, baseline_fitness: float = 0.0, candidates: list[Candidate] = <factory>, evaluated: int = 0) → None

all2md.optimize.content_tokens(text: str) → list[str]

Split text into words in a way that cannot be gamed by fragmenting them.

The objective’s text signal is a word count, so any setting that chops one word into two makes the document look like it contains more text. That is not a hypothetical: it has now bitten this objective three separate times.

consolidate_inline_formatting=False split runs so that “hello” was counted as “hel” + “lo”.
merge_hyphenated_words=False leaves a word broken across a line break, so repairing “hyphen-” + “ation” into “hyphenation” reduces the count by one – and the optimizer learned to recommend switching the repair off. It did so on 17 of 17 real papers, and ground truth confirmed the advice made every one of them worse.

Patching each setting as it appears does not work, because the defect is in the metric, not in the settings: a count of whitespace-separated tokens rewards fragmentation, so there is always another door. The fix is to count words the same way no matter how the parse happened to break them up – rejoin a token that ends in a hyphen with the one after it, and then drop hyphens entirely, so that

“hyphen-” + “ation” -> “hyphenation” “hyphenation” -> “hyphenation” “Anglo-” + “Saxon” -> “anglosaxon” “Anglo-Saxon” -> “anglosaxon”

all collapse to one identical token. A candidate that fragments text and one that does not now produce the same count, so the metric has no preference and cannot be gamed.

Note this deliberately conflates “well-known” with “wellknown”. That is fine: it does so for every candidate equally, and the count is only ever compared against other counts of the same document.

all2md.optimize.extract_metrics(document: Document) → DocumentMetrics

Measure the structural yield of a parsed document.

Reads the AST rather than the confidence report’s signal vector: measured on a two-column fixture the signals collapsed 6 option combinations into 2 distinct vectors (and the score into 1), while the AST kept 4 distinct outcomes. The signals are a summary built for a different purpose; the AST is the evidence.

all2md.optimize.find_furniture(texts: list[str], min_blocks: int = 2) → Furniture

Identify the repeated furniture in one parse of a document.

A sequence must appear in at least min_blocks distinct blocks – see furniture_threshold(). Counting distinct blocks (not occurrences) means a single block that repeats a phrase internally is never mistaken for furniture.

Repetition is the only reference-free evidence that content is furniture, so a one-page document offers no signal and its header cannot be recognized. That is a real limit of the objective, not a bug.

all2md.optimize.furniture_threshold(page_count: int | None) → int

How many distinct blocks a sequence must appear in before it is furniture.

Furniture is content that repeats on page after page, so the bar has to scale with the document. A flat “appears twice” rule is only correct for a two-page document – which is exactly the fixture it was written against. On a 21-page arXiv paper it fired on ordinary recurring academic phrasing and flagged 746 words of real body text as boilerplate, against 131 once the bar scaled with the page count. False furniture is not harmless: it makes body text look like clutter, so a setting that deletes that text scores as if it had tidied up.

all2md.optimize.score_candidates(candidates: list[Candidate]) → None

Assign each candidate a pool-relative fitness, in place.

Fitness is relative because the objective is reference-free: with no ground truth, the best text recovery observed across the pool is the closest thing to one. A candidate recovering fewer unique words than the best candidate lost text.

Dimensions the pool does not exercise are dropped and the weights renormalized over the rest — if no candidate found a table, the tables dimension says nothing about this document and must not drag every score down.

all2md.optimize.search(knobs: dict[str, list[Any]], evaluate: Callable[[dict[str, Any]], DocumentMetrics], *, presets: dict[str, dict[str, Any]] | None = None, rounds: int = 1) → OptimizationReport

Search knobs for the option set with the best fitness.

Takes an evaluate callable rather than a document, so the search is pure and can be tested against a synthetic objective with no parsing at all.

The shape is deliberately cheap. A full grid over the PDF knobs alone is tens of thousands of conversions; instead:

Score the stock defaults, then each named preset. These are interpretable, few, and often already contain the answer.
Coordinate descent from the best of those: walk one knob at a time, try each of its values holding the rest fixed, and keep any improvement. That is sum(len(values)) conversions per round rather than prod(len(values)).

Coordinate descent finds a local optimum, not a global one — it cannot discover that two knobs only pay off when flipped together. That is an accepted trade: the alternative costs orders of magnitude more conversions, and the knobs here are largely independent in practice. rounds > 1 re-walks the knobs from the new best point, which recovers some interactions.

Every distinct option set is evaluated at most once.

all2md.optimize.tunable_knobs(source_format: str) → dict[str, list[Any]]: Return the searchable option values for source_format (empty if untuned).