Website Intelligence Crawler

Website Intelligence Crawler: a public site as clean text, Markdown and chunks for embedding

Point it at a URL and it crawls the same-site pages and hands back each one as readable text, as Markdown, and as chunks already sized for an embedding model. Title, description, language, word count, the links it found and the status code ride along on every row.

No API key, no model call, nothing to authorise. The chunking and the relevance ranking are both plain arithmetic, so the same input gives the same output every time.

The one limit to understand first: there is no browser here. It reads the HTML the server sends. A page that builds its content in JavaScript after loading comes back with very little text, and no setting changes that. For ordinary documentation, blogs, marketing sites and knowledge bases, which is what most crawls are, the HTML is the content.

🕸️ What Website Intelligence Crawler does

From each URL you give it, it follows same-site links down to the depth you set, and writes a record per page. It stays on the site it started on, so an outbound link to somebody else's domain is recorded in links and not followed.

The text is the readable part of the page, with navigation, scripts and boilerplate stripped. The Markdown keeps the heading structure, paragraphs and lists, which is the version you want if the pages are going into a document store.

Chunks are cut to the size you ask for with a little overlap so a sentence is not sliced in half. Each one carries its character count and an estimated token count, so you can budget an embedding run before you start it.

Give it a query and every chunk gets a relevanceScore against those words. It is deterministic term matching, not a model, and it is there to let you keep the 20 chunks that matter instead of all 400.

robots.txt is honoured by default, including crawl delay.

📥 What you give it

{
  "startUrls": [{"url": "https://docs.example.com/"}],
  "maxPages": 50,
  "maxDepth": 2,
  "outputFormat": "both",
  "chunkSizeChars": 1200,
  "query": "pricing plans billing"
}

Leave the URLs empty and you get one sample row showing the shape, with nothing fetched and no rows charged.

📤 What you get back

A real row from a real run:

{
  "ok": true,
  "requestedUrl": "https://example.com/",
  "url": "https://example.com/",
  "seedUrl": "https://example.com/",
  "siteKey": "example.com",
  "depth": 0,
  "status": 200,
  "title": "Example Domain",
  "description": null,
  "language": "en",
  "canonicalUrl": "https://example.com/",
  "text": "Example DomainThis domain is for use in documentation examples without needing permission. Avoid use in operations.Learn more",
  "markdown": "# Example Domain\n\nThis domain is for use in documentation examples without needing permission. Avoid use in operations.\n\nLearn more",
  "links": [],
  "wordCount": 17,
  "contentType": "text/html",
  "chunks": [
    {
      "chunkIndex": 0,
      "text": "Example DomainThis domain is for use in documentation examples without needing permission. Avoid use in operations.Learn more",
      "charCount": 125,
      "estimatedTokens": 32,
      "relevanceScore": null
    }
  ],
  "query": null,
  "outputFormat": "both",
  "fetchedAt": "2026-09-06T17:08:09.247Z"
}

🧾 Reading the output

Page rows carry ok: true. Failures carry diagnostics: true with an errorCode, and are never charged.

One quirk of the table view: a failed row has no url, and the address it tried sits in requestedUrl, which the view does not show. Open the row or export the dataset to see which page it was.

▶️ How to run it

1. Open Website Intelligence Crawler and click Try for free. 2. Put your address into Start URLs. 3. Set Maximum pages and Maximum link depth. Ten pages at depth 1 is a good look before you commit to a whole site. 4. If the crawl is feeding a search or a RAG index, set Chunk size to whatever your embedding model likes. 5. Click Start, then download the dataset as JSON, CSV or Excel, or read it from the API.

💰 How much does it cost?

$0.60 per 1,000 pages, which is $0.0006 each. Flat on every Apify plan, no volume tiers.

You pay per page that came back. The sample row is not charged, and neither is a page that failed, was disallowed by robots.txt, was too large, or turned out to be a PDF. Those still arrive as rows so you know they happened.

Fifty pages is three cents. A hundred-page site crawled weekly is a quarter a month.

💡 What people use it for

• The dull half of a RAG pipeline: crawl the docs, get chunks with token counts, embed them.
• Keeping a support bot's knowledge current by re-crawling the help centre on a schedule.
• Content audits. wordCount, title, description and status across every page, in one export.
• Pulling a competitor's whole marketing site as Markdown to read properly.
• Finding the pages that actually mention a term, using query and relevanceScore.

🚧 What it does not do

• No browser. JavaScript-rendered content is not there to read.
• Same site only. Links to other domains are recorded, not followed.
• HTML only. PDFs, Word files and images come back as unsupported rather than as text.
• Nothing behind a login. No cookies, no sign-in, no interactive challenges.
• 100 pages per starting URL. For a bigger site, give it several starting URLs, one per section.
• No screenshots. The screenshot generator linked below does that.
• No summarising or tagging. No model runs here, which is the point: nothing is invented and

nothing is rephrased.

• Depth 0 is not honoured. It behaves as depth 1.

🧭 Which site tool do you need?

❓ Questions people ask

Does it run JavaScript? No. It reads the HTML the server sends, which is the whole content on most documentation and marketing sites.

Will it hammer my site? It keeps to robots.txt, including crawl delay, and you can drop maxConcurrency to 1.

What chunk size should I use? Whatever your embedding model is happiest with. 1,200 characters with 120 of overlap is a reasonable default, and estimatedTokens on each chunk tells you where you landed.

Can I crawl one page only? Set maxPages to 1. Depth 0 does not do it.

Do I get both text and Markdown? With outputFormat: both, yes, on the same row. Pick one if the rows are getting large.

Is crawling a public site legal? It reads public pages and honours robots.txt. Site terms still apply to what you do with the content afterwards. Apify's write-up on the legality of web scraping is a good starting point, and we are not lawyers.

🆘 If something breaks

Open the Issues tab on the actor page. Send the starting URL and the run ID. If a diagnostic row landed, its errorCode and requestedUrl usually name the reason already.