JavaScript Website Crawler

Feed your AI the pages that only fill in once their scripts run.

Crawl a website that builds its pages with JavaScript, and get every page as clean text, Markdown and chunks: each page is opened in a browser and read once its scripts have run, so the content a plain HTML read never sees is in the row. Title, description, headings, links, structured data and the status code come with every page. The catch is speed and price: a page takes from one second to nine in the browser, depending on how much script it runs, and on a site whose HTML already holds the text, a plain crawl does the same job faster and for less.

Input	Public web addresses to start from
Output	One row per page, as the browser saw it: text, Markdown, chunks with token estimates, title, description, headings, links, structured data, status
Ceiling	1,000 pages per start URL, 20 levels of links
Speed	About 1 to 5 seconds a page on most sites and 5 to 9 on script-heavy ones, at the default memory (measured on 29 October 2026)
Account needed	None

What JavaScript Website Crawler does

It starts at each URL you give it and follows links to other pages of the same site, down to the depth you set.

Every page is read twice. First the plain way, which is where robots.txt, redirects, the size limit and the check that it is an HTML page happen. Then in a browser that runs the page's scripts, waits for the page to settle and hands back what ended up on it. The row is built from that second read.

Some pages need more than a settle. Five settings cover that: wait for a CSS selector, wait a few seconds more, scroll to the bottom so content that loads on scroll arrives, open collapsed sections such as FAQ answers, and skip images, fonts and styles so pages load faster.

Every row carries needsJavaScript, judged on the HTML the server sent before any script ran. It tells you which pages really needed the browser. If most of a site comes back false, Website Intelligence Crawler reads it without one.

The text, Markdown and chunks work as in that crawler: boilerplate out, headings kept, chunks cut to your size with a little overlap and an estimated token count.

What data you get from each page

What you get	Field
The page's readable text, after its scripts ran	`text`
The same content as Markdown, headings kept	`markdown`
Chunks with character count, token estimate and relevance score	`chunks`
Title, description, language and canonical address	`title`, `description`, `language`, `canonicalUrl`
Same-site links with their anchor text, the ones scripts wrote included	`links`
Word count and HTTP status	`wordCount`, `status`
Where it came from: the start URL, the site, the link depth and the page that linked here	`seedUrl`, `siteKey`, `depth`, `referrerUrl`
The address asked for, and the one the page ended on	`requestedUrl`, `url`, `finalUrl`
Whether the page was an empty shell until its scripts ran	`needsJavaScript`
Time the server took, and time the browser spent	`responseTimeMs`, `renderTimeMs`
Open Graph, site name, sharing image, published date, keywords	`openGraph`, `siteName`, `imageUrl`, `publishedAt`, `keywords`
Headings with their level	`headings`
Links to other websites, and link totals by place on the page	`externalLinks`, `linkStats`
Robots tag, noindex and where the canonical points	`indexability`
Structured data: its types and the JSON-LD blocks	`schemaTypes`, `jsonLd`
Images with alt text and declared size	`images`, `imageStats`
Paragraph and sentence counts, reading ease, top words	`contentStats`
Response headers	`responseHeaders`
Pages in the run sharing a title or description	`duplicateTitleOf`, `duplicateDescriptionOf`
Optional: the cleaned HTML of the content	`html`

What it does not do

Nothing behind a login. No cookies, no sign-in, no forms filled.
A page the site will not serve comes back as a free note, not a page.
One site at a time. Links to other websites are listed in externalLinks but not followed.
HTML pages only. PDFs, office files and images come back as notes rather than text.
No screenshots and no clicking around. It reads pages; the only clicks are the collapsed-section ones you ask for.
No summarising. No model runs here, so nothing is rephrased or invented.
It is not an official tool of any platform or company named on this page. Dami's Studio is independent and is not affiliated with or endorsed by any of them.

How it works

1
Open it on Apify
The Open on Apify button takes you there. Nothing to install.
2
Set the inputs
The form opens with an example in Start URLs. Put in your own and check the other settings.
3
Click Run
It runs on Apify's servers, on your own Apify account.
4
Export the results
Download them from the run, or read them over Apify's API.

Pricing

$0.002 per rendered page = $2 per 1,000

Live price on Apify's Free plan, read on 30 October 2026. Every plan pays the same. You are billed on your own Apify account. The full price list is on the Apify page.

Use it from your AI chat

Add Apify's connector to your AI chat once (setup steps), then paste this:

Use the JavaScript Website Crawler (dami_studio/javascript-website-crawler) to crawl "https://example.com/", and return one row per page, as the browser saw it: text, Markdown, chunks with token estimates, title, description, headings, links, structured data, status.

Some apps ask for a server address instead. This one loads just this tool:

https://mcp.apify.com?tools=dami_studio/javascript-website-crawler

Every run your AI chat starts goes on your own Apify account, at the prices on this tool's Apify page.

Inputs

Field	What it does	Type
`startUrls`	Public web addresses to start from. Links to other pages of the same site are followed from each one.	array
`maxPages`	How many pages to read from each start URL, pages that could not be read included. The run's maximum cost can stop it sooner.	integer
`maxDepth`	How many clicks away from the start URL to go. 0 reads only the start pages.	integer
`maxPagesPerRun`	A page limit shared by all start URLs together. Leave it empty to keep only the limit per start URL.	integer
`maxConcurrency`	How many pages of one site are worked on at the same time.	integer
`useRobotsTxt`	Honour the site's robots.txt: paths it disallows are not read, and its crawl delay is kept (10 seconds at most).	boolean
`requestTimeoutSecs`	Seconds the first read of a page may take. The browser then gets twice this, between 20 and 60 seconds, to load the page.	integer
`maxResponseSizeKb`	The biggest page accepted, in kilobytes. A page that grows past twice this (1 MB at least) once its scripts have run is refused too, as a free note.	integer
`waitForSelector`	Wait up to 5 seconds for an element matching this CSS selector before reading the page, for example .product-list. A page where it never appears comes back as a free SELECTOR_NOT_FOUND note.	string
`renderWaitSecs`	Wait this long after the page has settled before reading it, 0 to 5.	integer
`scrollToBottom`	Scroll to the bottom a few times, so content that only loads as you scroll is there.	boolean
`expandSections`	Open details boxes and buttons marked as collapsed, such as FAQ answers. Links, forms and send buttons are never clicked.	boolean
`blockMedia`	Don't load images, video, fonts or stylesheets, so pages load faster. If a page comes back with less text with it on, run that page without it.	boolean
`includeUrlGlobs`	Follow a link only when it matches one of these patterns. `` matches inside one folder, `` across folders. A pattern starting with / is matched against the path and query (`/docs/*`), a full address against the whole URL. Start URLs are always read. Up to 20 patterns.	array
`excludeUrlGlobs`	Never follow a link that matches one of these patterns, for example `/tag/` or `?page=*`. Same pattern rules as above.	array
`stayUnderStartPath`	Only follow links inside the start URL's folder: a crawl started at /docs/ stays in /docs/ and leaves /docs-old/ alone.	boolean
`includeSubdomains`	Treat subdomains of the start site as the same site, so a crawl of example.com also reads docs.example.com and help.example.com.	boolean
`followNavigationLinks`	Also follow links in menus, headers and footers. Without it only links in the body of the page are followed, which can leave a docs site with a sidebar at one page.	boolean
`skipFileLinks`	Don't request links that end in a file type: PDFs, office files, images, media, archives, scripts, styles and feeds.	boolean
`countOnlyGoodPages`	Pages that could not be read don't count toward Pages per start URL. The crawl still stops after twice that many tries.	boolean
`useSitemap`	Also start from the pages in the site's sitemap (named in robots.txt, or /sitemap.xml), sitemap indexes included. A start URL ending in .xml is read as a sitemap.	boolean
`requestLanguage`	The language list sent with each page, for sites that answer in the visitor's language, for example de-DE,de;q=0.9. Empty means US English.	string
`outputFormat`	Keep clean text, Markdown, or both in each page row.	string
`markdownStyle`	Basic keeps headings, paragraphs and lists. Rich also keeps links, bold and italics, tables and code blocks. Chunks are always cut from the plain text.	string
`contentExtractor`	Main content reads the page's main or article element, or the body when there is none. Article reader picks the block with the most paragraph text. Whole page reads the body.	string
`keepSelector`	Keep only the parts of the page that match this CSS selector, for example article .content. When nothing matches, the normal text is kept and keepSelectorMatched is false.	string
`removeSelector`	Remove the parts of the page that match this CSS selector before the text is read, for example .sidebar, .newsletter. Links are still followed from them.	string
`removeCookieBanners`	Remove cookie and consent dialogs from the text, so their wording stays out of the first chunk.	boolean
`dropRepeatedLines`	Drop paragraphs and other text blocks already seen on an earlier page of the same site, such as sidebars and sign-up boxes. repeatedLinesDropped counts what went.	boolean
`includeHtml`	Add the cleaned HTML of the content to each row as html, as it stood after the scripts ran. Rows get several times bigger.	boolean
`query`	Words that score each chunk's relevance, so you can keep the best chunks for a question. Plain word matching, no AI.	string
`chunkSizeChars`	The most characters in each chunk of text.	integer
`chunkOverlapChars`	Characters repeated between neighbouring chunks, so a sentence cut in two still reads in one of them.	integer
`saveSkippedUrls`	Save the links your rules left out, with the reason, in the key-value store record SKIPPED_URLS. Up to 10,000.	boolean

The full input form is on the Apify page.

Questions people ask

How much does JavaScript Website Crawler cost?

$0.002 per rendered page ($2 per 1,000) on Apify's Free plan. Every plan pays the same. It is billed on your own Apify account.

Can I run JavaScript Website Crawler from my AI chat?

Yes, from any app that supports MCP connectors. Add Apify's connector (https://mcp.apify.com/), sign in with your Apify account, and ask for dami_studio/javascript-website-crawler. The steps are on the Use with AI page. Each run goes on your own Apify account, at the prices on its Apify page.

Do I need to write code to use JavaScript Website Crawler?

No. Open it on Apify, fill in the form, click Run and download the results.