# JavaScript Website Crawler

One of the tools on damistudio.io. Index of all of them: https://damistudio.io/llms.txt

URL: https://damistudio.io/tools/javascript-website-crawler/
Run: https://apify.com/dami_studio/javascript-website-crawler?fpr=wcqem0
Category: Website & List Checks
Pricing: $0.002 per rendered page ($2 per 1,000) on Apify's Free plan, read on 5 November 2026.
Apify name: dami_studio/javascript-website-crawler
Latest version: 0.1.10, updated 4 November 2026
From an AI chat: add Apify's connector (https://mcp.apify.com/), sign in with your Apify account, then ask:
  Use the JavaScript Website Crawler (dami_studio/javascript-website-crawler) to crawl "https://example.com/", and return one row per page, as the browser saw it: text, Markdown, chunks with token estimates, title, description, headings, links, structured data, status.
Preset address for apps that take one: https://mcp.apify.com?tools=dami_studio/javascript-website-crawler (runs go on your own Apify account)

Crawl a public website in a browser and get each page as it looks after its scripts run: clean text, Markdown, links, metadata and chunks.

Inputs:
- `startUrls` (array, required): Public web addresses to start from. Links to other pages of the same site are followed from each one.
- `maxPages` (integer): How many pages to read from each start URL, pages that could not be read included. The run's maximum cost can stop it sooner.
- `maxDepth` (integer): How many clicks away from the start URL to go. 0 reads only the start pages.
- `maxPagesPerRun` (integer): A page limit shared by all start URLs together. Leave it empty to keep only the limit per start URL.
- `maxConcurrency` (integer): How many pages of one site are worked on at the same time.
- `useRobotsTxt` (boolean): Honour the site's robots.txt: paths it disallows are not read, and its crawl delay is kept (10 seconds at most).
- `requestTimeoutSecs` (integer): Seconds the first read of a page may take. The browser then gets twice this, between 20 and 60 seconds, to load the page.
- `maxResponseSizeKb` (integer): The biggest page accepted, in kilobytes. A page that grows past twice this (1 MB at least) once its scripts have run is refused too, as a free note.
- `waitForSelector` (string): Wait up to 5 seconds for an element matching this CSS selector before reading the page, for example .product-list. A page where it never appears comes back as a free SELECTOR_NOT_FOUND note.
- `renderWaitSecs` (integer): Wait this long after the page has settled before reading it, 0 to 5.
- `scrollToBottom` (boolean): Scroll to the bottom a few times, so content that only loads as you scroll is there.
- `expandSections` (boolean): Open details boxes and buttons marked as collapsed, such as FAQ answers. Links, forms and send buttons are never clicked.
- `blockMedia` (boolean): Don't load images, video, fonts or stylesheets, so pages load faster. If a page comes back with less text with it on, run that page without it.
- `includeUrlGlobs` (array): Follow a link only when it matches one of these patterns. `*` matches inside one folder, `**` across folders. A pattern starting with / is matched against the path and query (`/docs/**`), a full address against the whole URL. Start URLs are always read. Up to 20 patterns.
- `excludeUrlGlobs` (array): Never follow a link that matches one of these patterns, for example `/tag/**` or `**?page=*`. Same pattern rules as above.
- `stayUnderStartPath` (boolean): Only follow links inside the start URL's folder: a crawl started at /docs/ stays in /docs/ and leaves /docs-old/ alone.
- `includeSubdomains` (boolean): Treat subdomains of the start site as the same site, so a crawl of example.com also reads docs.example.com and help.example.com.
- `followNavigationLinks` (boolean): Also follow links in menus, headers and footers. Without it only links in the body of the page are followed, which can leave a docs site with a sidebar at one page.
- `skipFileLinks` (boolean): Don't request links that end in a file type: PDFs, office files, images, media, archives, scripts, styles and feeds.
- `countOnlyGoodPages` (boolean): Pages that could not be read don't count toward Pages per start URL. The crawl still stops after twice that many tries.
- `useSitemap` (boolean): Also start from the pages in the site's sitemap (named in robots.txt, or /sitemap.xml), sitemap indexes included. A start URL ending in .xml is read as a sitemap.
- `requestLanguage` (string): The language list sent with each page, for sites that answer in the visitor's language, for example de-DE,de;q=0.9. Empty means US English.
- `outputFormat` (string): Keep clean text, Markdown, or both in each page row.
- `markdownStyle` (string): Basic keeps headings, paragraphs and lists. Rich also keeps links, bold and italics, tables and code blocks. Chunks are always cut from the plain text.
- `contentExtractor` (string): Main content reads the page's main or article element, or the body when there is none. Article reader picks the block with the most paragraph text. Whole page reads the body.
- `keepSelector` (string): Keep only the parts of the page that match this CSS selector, for example article .content. When nothing matches, the normal text is kept and keepSelectorMatched is false.
- `removeSelector` (string): Remove the parts of the page that match this CSS selector before the text is read, for example .sidebar, .newsletter. Links are still followed from them.
- `removeCookieBanners` (boolean): Remove cookie and consent dialogs from the text, so their wording stays out of the first chunk.
- `dropRepeatedLines` (boolean): Drop paragraphs and other text blocks already seen on an earlier page of the same site, such as sidebars and sign-up boxes. repeatedLinesDropped counts what went.
- `includeHtml` (boolean): Add the cleaned HTML of the content to each row as html, as it stood after the scripts ran. Rows get several times bigger.
- `query` (string): Words that score each chunk's relevance, so you can keep the best chunks for a question. Plain word matching, no AI.
- `chunkSizeChars` (integer): The most characters in each chunk of text.
- `chunkOverlapChars` (integer): Characters repeated between neighbouring chunks, so a sentence cut in two still reads in one of them.
- `saveSkippedUrls` (boolean): Save the links your rules left out, with the reason, in the key-value store record SKIPPED_URLS. Up to 10,000.

Documentation:

Give it a website's address and it opens each page in a real browser, waits for the page's scripts to finish, then reads what's there: the text block by block, Markdown, chunks sized for an AI index, the links, the headings, the metadata and any structured data. It stays on the site you gave it. Every row also says whether that page actually needed JavaScript, so you can tell which crawler a site deserves.

## Who needs a JavaScript crawler

- **Teams loading documentation into an AI search** when the docs are built as an app and a plain fetch comes back with an empty shell.
- **Checking your own site the way a search engine renders it.** `needsJavaScript` marks the pages whose text only shows up after scripts run.
- **Help centres with folded answers.** `expandSections` opens them before the page is read.

## A worked example: this site

On 4 November we pointed it at damistudio.io's tools page with a three-page limit. It read the tools page and two pages linked from it in 24 seconds, spending between one and two seconds of browser time on each.

The tools page came back as 1,273 words in 8 chunks, a category page as 298 words, a tool page as 1,178. `needsJavaScript` was false on all three: this site is plain HTML, so the lighter [Website Intelligence Crawler](https://damistudio.io/tools/website-intelligence-crawler/) would have done the same job faster. A last, free row summed up the crawl.

You pay per page read.

## What a row looks like

| Field | What it tells you |
|---|---|
| `url`, `status`, `title`, `language` | The page, its status code, its title and language |
| `text`, `markdown` | The content as plain text and as Markdown |
| `chunks` | The text cut into pieces with `chunkIndex` and `charCount`, ready for an AI index |
| `headings`, `links`, `externalLinks` | The page's outline, its links on the site, and those leaving it |
| `jsonLd`, `schemaTypes` | Structured data the page carries |
| `needsJavaScript`, `renderTimeMs` | Whether the text depended on scripts, and how long the page took to render |
| `wordCount` | How much text it found |

## When it's the wrong tool

- **The site's HTML already holds its text.** [Website Intelligence Crawler](https://damistudio.io/tools/website-intelligence-crawler/) reads it without a browser.
- **You want a site's DNS, certificate and headers rather than its pages.** That's [Domain Inspector](https://damistudio.io/tools/domain-inspector/).
- **You want encyclopedia articles as text.** That's [Wikipedia Scraper](https://damistudio.io/tools/wikipedia-scraper/).
- **The pages sit behind a sign-in.** It reads public pages only.
- It is not an official tool of any platform or company named on this page. Dami's Studio is independent and is not affiliated with or endorsed by any of them.

## Questions people ask

### What is a JavaScript crawler?

A crawler that loads each page in a browser and lets its scripts run before reading it, so content that only appears once the page has loaded ends up in the row.

### How do I crawl a website that uses JavaScript?

Put the address in `startUrls` and a page limit in `maxPages`. For pages that fill in slowly, `renderWaitSecs` waits up to five seconds more, `waitForSelector` waits for a particular element, and `scrollToBottom` triggers content that loads on scroll.

### Can I export a website to Markdown?

Yes. Each page it reads comes back as Markdown and as plain text, with the chunks alongside.

### Does it follow links to other websites?

No. It stays on the site you started from, and the outside links are listed in `externalLinks`.

### What does JavaScript Website Crawler do?

Crawl a public website in a browser and get each page as it looks after its scripts run: clean text, Markdown, links, metadata and chunks.

### What do I give JavaScript Website Crawler?

Public web addresses to start from.

### What do I get back from JavaScript Website Crawler?

One row per page, as the browser saw it: text, Markdown, chunks with token estimates, title, description, headings, links, structured data, status.

### How much does JavaScript Website Crawler cost?

$0.002 per rendered page ($2 per 1,000) on Apify's Free plan. Every plan pays the same. It is billed on your own Apify account.

### Can I run JavaScript Website Crawler from my AI chat?

Yes, from any app that supports MCP connectors. Add Apify's connector (https://mcp.apify.com/), sign in with your Apify account, and ask for dami_studio/javascript-website-crawler. The steps are on the Use with AI page. Each run goes on your own Apify account, at the prices on its Apify page.

### Do I need to write code to use JavaScript Website Crawler?

No. Open it on Apify, fill in the form, click Run and download the results.
