#markitdown-mb

InputSource
  -> FormatDetector
  -> ParserRegistry
  -> ParseResult
  -> runtime / IRInput lowering
  -> pipeline passes
  -> RenderInput
  -> Renderer
  -> Markdown / debug JSON

#Why This Project

• Performance: official benchmarking uses bench v2, which measures only release binaries and real product paths. For formats that better represent the complex document pipeline, reproduced official compare results currently show:

  Format	CLI speedup vs markitdown
  docx	88.09x
  xlsx	30.75x
  epub	18.07x
  pdf	30.60x

  For txt, csv, tsv, and json, the implementation is not trading away semantics or diagnostics just to widen benchmark gaps. It still keeps the full product-path information and output constraints, so the margin is often less dramatic, while remaining faster under the formal benchmark definition.
• Medium-to-large files and stress cases: the unified v2 path is more stable on larger inputs. On some stress samples where the external baseline cannot form a complete comparable set, the MoonBit path can still complete conversion successfully. These claims can be reproduced directly with bench v2 compare runs on specific rows.
• Memory profile: in the current tracked benchmark samples, the median peak RSS for moonbit-cli is approximately 48,328 KB, compared with 176,640 KB for markitdown. Public-facing summaries still prioritize performance and trust guarantees; for full memory details, inspect the run artifacts directly.
• Stability: benchmarking includes provenance, route coverage, and fidelity gates. Missing provenance, route mismatches, or route_fidelity_status != matched immediately invalidate trust for the MoonBit case.
• Clear boundaries: unsupported formats fail closed. The product does not rely on hidden fallback paths to present a false success surface.

#Support Matrix

• txt
• csv
• tsv
• json
• jsonl
• ndjson
• xml
• yaml
• yml
• html
• htm
• markdown
• md
• zip
• epub
• docx
• xlsx
• pptx
• pdf

• pdf is officially supported only for native-text PDFs.
• Scanned or image-only PDFs still fail closed.
• pdf --ocr is not supported.
• Default image input is not part of the formal support matrix, but explicit image --ocr can use local Tesseract.
• Unsupported formats fail closed.

#Quick Start

moon build cli --target native

./_build/native/debug/build/cli/cli.exe --help

./_build/native/debug/build/cli/cli.exe normal samples/main_process/txt/markdown/txt_plain.txt .tmp/manual/out.md

./_build/native/debug/build/cli/cli.exe normal samples/main_process/pdf/markdown/root_native_text_baseline.pdf .tmp/manual/pdf.md

#OCR

brew install tesseract
brew install tesseract-lang

• --ocr is available only for the explicit image OCR boundary.
• --ocr-lang <LANG> can be used to specify language.
• pdf --ocr is not currently enabled.
• Scanned/image-only PDFs remain fail-closed.

#Reproducibility Guide

moon fmt
moon info
moon check
moon build
moon test
bash samples/check.sh
bash samples/check.sh --check-inventory
bash samples/helpers/contracts/check_root_contracts.sh

bash samples/check_quality.sh
bash samples/check_quality.sh --format pdf

moon build --target native --release --package ZSeanYves/markitdown/cli
moon build --target native --release --package ZSeanYves/markitdown/bench/runner
_build/native/release/build/bench/runner/runner.exe doctor
_build/native/release/build/bench/runner/runner.exe run --preset official-internal
_build/native/release/build/bench/runner/runner.exe run --preset official-compare --markitdown-path /path/to/markitdown
_build/native/release/build/bench/runner/runner.exe run --scenario compare.official_compare --bench-id docx_medium_golden_v1 --markitdown-path /path/to/markitdown
_build/native/release/build/bench/runner/runner.exe run --scenario compare.official_compare --bench-id xlsx_medium_regional_1971_2020_v1 --markitdown-path /path/to/markitdown
_build/native/release/build/bench/runner/runner.exe run --scenario compare.official_compare --bench-id epub_medium_alice_v1,pdf_medium_nist_800_207_v1 --markitdown-path /path/to/markitdown
_build/native/release/build/bench/runner/runner.exe run --scenario compare.official_compare --bench-id json_medium_spdx_licenses_v1 --markitdown-path /path/to/markitdown

#Project Structure

• ParserRegistry selects parsers; it does not own final rendering.
• Every parser returns ParseResult.
• IRInput and RenderInput are the stable cross-layer product shapes.
• Renderer owns final Markdown output.
• format_readers must not depend on runtime, pipeline, render, or convert.

#Samples And Quality Lab

• samples/main_process/ contains lightweight repo-local regression samples and expected outputs.
• samples/check.sh is the primary sample regression entrypoint.
• samples/helpers/contracts/check_root_contracts.sh is the aggregated contract entrypoint.
• markitdown-quality-lab/ is the local external corpus repository for quality and benchmark validation.

• EPUB support is implemented through format_readers/epub on top of format_readers/zip.
• ZIP archive reading continues to rely on bikallem/compress/flate inside format_readers/zip.