Document parsing accuracy benchmark

For each labeled region we gather the predicted boxes overlapping its location (page-sized dumps are excluded), stitch their text in reading order, and measure how much of the region's text is recoverable in that local context — $1$ when the text is contained, otherwise the best-matching window scored by $1-\mathrm{NED}$. The metric averages this recall $\frac{1}{N}\sum_{r}\mathrm{rec}_r$ over all $N$ regions, with a region that got no overlapping prediction scoring 0; extra neighboring text from a coarse box is not penalized, so word-, line-, and paragraph-box outputs are scored fairly.

Character fidelity over the whole page in each provider's emitted reading order: $1-\mathrm{NED}$, where $\mathrm{NED}=\frac{\text{edit distance}}{\max(|p|,\,|g|)}$ is the Levenshtein distance between prediction $p$ and ground truth $g$ over the longer length. Markdown, case, and Textract LAYOUT table scaffolding are normalized first.

Bag-of-words overlap, independent of order: the harmonic mean of word precision $P$ and recall $R$, $F_1=\frac{2PR}{P+R}$. It surfaces which words were captured versus dropped or hallucinated.

How well the predicted boxes cover the real content blocks: intersection-over-union of the predicted ($A$) and verified ($B$) box masks rasterized on a 1000×1000 grid, $\frac{|A\cap B|}{|A\cup B|}$, independent of the text inside.