Turn PDFs and Office files into Markdown with page-level citations

An agent cannot cite a page it cannot locate

The Document to Markdown endpoint converts a PDF (text layer), DOCX, XLSX or PPTX file, or normalizes HTML, Markdown, text, CSV or JSON, into Markdown or plain text. It returns one unit per page, sheet or slide with its exact character range and SHA-256, so a quote can be tied to "page 7, characters 4120 to 5890, hash …" instead of "somewhere in the PDF". Scanned pages that have no text layer are flagged with needs_ocr rather than silently returned empty.

Convert a document

Give an https URL, or up to 4 MB of the file as content_base64 with a filename. pages selects pages, sheets or slides (for example "1-5,8", up to 200; the default is the first 200), max_chars caps the output, and format is markdown or text.

curl -sS -X POST "https://$RAPIDAPI_HOST/v1/doc-to-markdown" \
  -H "Content-Type: application/json" \
  -H "X-RapidAPI-Key: $RAPIDAPI_KEY" \
  -H "X-RapidAPI-Host: $RAPIDAPI_HOST" \
  -d '{"url": "https://arxiv.org/pdf/1706.03762", "pages": "1", "max_chars": 1500}' > doc.json

jq '.data | {document_kind, units_total, metadata, units, truncated}' doc.json

What a unit looks like

For the 15-page paper above the response reported document_kind: pdf, units_total: 15, the file's size and SHA-256, and for the one selected page:

{
  "index": 1,
  "label": "page 1",
  "char_start": 0,
  "char_end": 1500,
  "sha256": "953cc513bcd229e9de90165866b64393bbe28a8900f9173118d4b007ebfff2bf",
  "needs_ocr": false,
  "truncated": true
}

{ "truncated": true, "notes": ["Output cut at 1500 characters after 1 of 1 selected unit(s)."], "limits": { "max_bytes": 4194304 } }

Store a citation, not just text

Keep the document URL, the document hash from source.sha256, and for each passage the unit index, char_start, char_end and unit sha256. A reader can later fetch the file, check the hashes and land on the exact page. char_start and char_end are UTF-16 code unit offsets into content, which is how JavaScript indexes strings; in Python, encode to UTF-16 before slicing.

Some hosts reject automated downloads. A public W3C test PDF we tried returned fetch_failed with an HTTP 403, and another file returned 404. When the URL fails, download the file yourself and send it as content_base64. Scanned documents: check needs_ocr on each unit and run OCR only on those pages; the endpoint reads the text layer and does not OCR images.

import base64, os, requests

def convert(path):
    r = requests.post(
        f"https://{os.environ['RAPIDAPI_HOST']}/v1/doc-to-markdown",
        headers={"X-RapidAPI-Key": os.environ["RAPIDAPI_KEY"], "X-RapidAPI-Host": os.environ["RAPIDAPI_HOST"]},
        json={"content_base64": base64.b64encode(open(path, "rb").read()).decode(), "filename": os.path.basename(path)},
        timeout=120,
    )
    r.raise_for_status()
    return r.json()["data"]

doc = convert("contract.pdf")
for u in doc["units"]:
    if u["needs_ocr"]:
        print("needs OCR:", u["label"])
        continue
    units16 = doc["content"].encode("utf-16-le")
    text = units16[u["char_start"] * 2:u["char_end"] * 2].decode("utf-16-le")
    print({"doc": doc["source"]["sha256"][:12], "unit": u["label"], "range": [u["char_start"], u["char_end"]], "unit_sha256": u["sha256"][:12], "chars": len(text)})