Website Intelligence Crawler - Text, Markdown & RAG Chunks
Feed your AI a public website, page by page.
Give it a site's address and it reads the pages the server sends, following links on the same site, and turns each page into a row: clean text, Markdown, chunks sized for an embedding model, and the title, description, language, word count, links and status code. No browser runs and no AI model is called, so it's quick and nothing gets rephrased.
Who turns websites into Markdown
- Teams building an AI assistant on their own docs. The chunks go straight into an embedding index, each one tied to its page's address.
- Moving a site to a new system. The pages as Markdown with their titles and links, without copying and pasting.
- Content audits. Title, description and word count for each page in one table, and
duplicateTitleOfpoints out pages sharing a title.
A worked example: this site
On 4 November we gave it damistudio.io's tools page with a three-page limit. The run took about two seconds, with each page answering in 121 to 196 milliseconds.
The tools page came back as 1,141 words in 8 chunks, a category page as 264, a tool page as 1,079. needsJavaScript was false on all three. The same three pages took 24 seconds in the JavaScript crawler, which a site like this doesn't need.
You pay per page.
What a row looks like
| Field | What it tells you |
|---|---|
url, status, title, description, language | The page and what it says about itself |
text, markdown | The readable content, as plain text and as Markdown |
chunks | The text in pieces, each with charCount, estimatedTokens, and a relevanceScore when you give a query |
links | Links to other pages on the same site |
wordCount | How much text the page holds |
needsJavaScript | Whether the page looks like it builds its text with scripts |
duplicateTitleOf, duplicateDescriptionOf | Another page in the crawl with the same title or description |
When it's the wrong tool
- The site builds its pages with JavaScript. JavaScript Website Crawler reads them after the scripts have run.
- You want a domain's DNS, certificate and security headers. That's Domain Inspector.
- You want the pages as they were years ago. That's Wayback Machine Scraper.
- It is not an official tool of any platform or company named on this page. Dami's Studio is independent and is not affiliated with or endorsed by any of them.
How it works
- 1Open it on Apify
The Open on Apify button takes you there. Nothing to install.
- 2Set the inputs
The form opens with an example in Start URLs. Put in your own and check the other settings.
- 3Click Run
It runs on Apify's servers, on your own Apify account.
- 4Export the results
Download them from the run, or read them over Apify's API.
Pricing
$0.0006 per page = $0.60 per 1,000
Live price on Apify's Free plan, read on 5 November 2026. Paid plans pay a little less. You are billed on your own Apify account. The full price list is on the Apify page.
Use it from your AI chat
Add Apify's connector to your AI chat once (setup steps), then paste this:
Use the Website Intelligence Crawler (dami_studio/website-intelligence-crawler) to crawl "https://example.com/", and return one row per page: readable text, Markdown, chunks with token estimates, title, description, language, links, status.Some apps ask for a server address instead. This one loads just this tool:
https://mcp.apify.com?tools=dami_studio/website-intelligence-crawlerEvery run your AI chat starts goes on your own Apify account, at the prices on this tool's Apify page.
Questions people ask
How do I convert a website to Markdown?
Put the address in startUrls, set maxPages and maxDepth for how far it should go, and leave outputFormat on both or set it to markdown.
Can I convert just one page from HTML to Markdown?
Yes. Give it that page's address and set maxPages to 1.
Are the chunks ready for embeddings?
Yes. Each page's text is cut into pieces of about 1,200 characters, overlapping by 120, each with an estimated token count. chunkSizeChars and chunkOverlapChars change both.
Does it respect robots.txt?
Yes. A page the site's robots.txt rules out comes back as a free note instead of a page.
How much does Website Intelligence Crawler cost?
$0.0006 per page ($0.60 per 1,000) on Apify's Free plan. Paid plans pay a little less. It is billed on your own Apify account.
Can I run Website Intelligence Crawler from my AI chat?
Yes, from any app that supports MCP connectors. Add Apify's connector (https://mcp.apify.com/), sign in with your Apify account, and ask for dami_studio/website-intelligence-crawler. The steps are on the Use with AI page. Each run goes on your own Apify account, at the prices on its Apify page.
Do I need to write code to use Website Intelligence Crawler?
No. Open it on Apify, fill in the form, click Run and download the results.