Give an AI agent clean Markdown from any URL and detect when it changes

Raw HTML wastes the context window

A fetched web page is mostly navigation, scripts and markup. The Web Page to Clean Markdown endpoint fetches one public page and returns the main content as GitHub-flavored Markdown or plain text, with the title, canonical URL, language, absolute links and two hashes. It uses no model, so the same page gives the same output.

curl -sS -X POST "https://$RAPIDAPI_HOST/v1/page-contents" \
  -H "Content-Type: application/json" \
  -H "X-RapidAPI-Key: $RAPIDAPI_KEY" \
  -H "X-RapidAPI-Host: $RAPIDAPI_HOST" \
  -d '{"url": "https://example.com", "format": "markdown", "include_links": true}' \
  | jq '.data | {title, final_url, lang, word_count, content, content_sha256}'

What you get back

For https://example.com the response held the title "Example Domain", language en, a word count of 25, the extracted text, content_sha256 (0a080719…d00f) and html_sha256. Options: format (markdown or text), max_chars (default 60000), include_links and include_tables (both default true).

Know the limits before you build on it. The endpoint does not run JavaScript, so a page rendered on the client comes back sparse; it makes one request and does not follow links; and it does not consult robots.txt, so respect a site's terms yourself. Use truncated to know whether max_chars cut the content.

Detect a change without storing the page

content_sha256 hashes the extracted content and html_sha256 hashes the fetched HTML. Compare content_sha256 between two fetches to ignore markup-only changes, such as a new tracking script, while still catching an edit to the text. Store only the hash and the URL.

import json, os, pathlib, requests

STATE = pathlib.Path("page-hashes.json")
seen = json.loads(STATE.read_text()) if STATE.exists() else {}

def fetch(url):
    r = requests.post(
        f"https://{os.environ['RAPIDAPI_HOST']}/v1/page-contents",
        headers={"X-RapidAPI-Key": os.environ["RAPIDAPI_KEY"], "X-RapidAPI-Host": os.environ["RAPIDAPI_HOST"]},
        json={"url": url}, timeout=60,
    )
    r.raise_for_status()
    return r.json()["data"]

for url in ["https://example.com/pricing", "https://example.com/terms"]:
    page = fetch(url)
    if seen.get(url) not in (None, page["content_sha256"]):
        print("changed:", url)
    seen[url] = page["content_sha256"]

STATE.write_text(json.dumps(seen, indent=2))

Feed it to a model

Use final_url (after redirects) or canonical as the citation, and pass the content into the prompt with the hash alongside it, so an answer can be tied to the exact text it was based on. For pages that need JavaScript to render, fetch the rendered HTML yourself and send it to the Document to Markdown endpoint instead, which accepts HTML as a document.