arXiv Scraper

arXiv Scraper: papers, full abstracts and PDF links for any arXiv search

Write an arXiv query and get one row per paper: the id, title, the whole abstract, every author, the categories, both dates and direct links to the abstract page and the PDF. No account, no API key.

Metadata and links only. The PDF link is there for you to fetch yourself; this actor does not download or read the paper.

🔬 What arXiv Scraper does

It runs your query against arXiv and pages through the results, pausing between pages because arXiv asks callers to. Duplicate papers across pages are dropped by id.

The query uses arXiv's own syntax, which is worth two minutes of your time because it is what makes this precise:

Combine them with AND, OR and ANDNOT. ti:attention AND au:vaswani does exactly what it looks like. Sort by relevance, by submission date or by last revision, newest first in both date orders.

📥 What you give it

{
  "query": "all:large language models",
  "sortBy": "relevance",
  "maxItems": 50
}

There is no date-range field. To narrow by time, sort on submittedDate and cut the rows on publishedAt yourself.

📤 What you get back

A real row from a recent run, with the abstract cut short:

{
  "ok": true,
  "charged": true,
  "arxivId": "2609.13144",
  "title": "Type Diversity Enables Transformers to Generalise Compositionally",
  "abstract": "Compositional generalisation has been divided into lexical and structural generalisation. Previous work has found that structural generalisation is harder than lexical for Transformers...",
  "authors": ["Anssi Moisio", "Mathias Creutz", "Mikko Kurimo"],
  "primaryCategory": "cs.CL",
  "categories": ["cs.CL"],
  "publishedAt": "2026-09-11T00:00:00.000Z",
  "updatedAt": "2026-09-11T00:00:00.000Z",
  "doi": null,
  "absUrl": "https://arxiv.org/abs/2609.13144",
  "pdfUrl": "https://arxiv.org/pdf/2609.13144"
}

🧾 Reading the output

Two kinds of row can land in your dataset.

charged: true is simply the marker every paper row carries. Use it, or ok, to tell results from diagnostics. Do not read it as a record of what your account was billed.

Rows arrive at the end. Everything is collected before anything is written, so a large run shows an empty dataset until it finishes. Keep a first run small while you get the query right.

The default table view has no ok column, so a diagnostic row renders as one blank line. Switch to All fields or export as JSON.

▶️ How to run it

1. Open arXiv Scraper and click Try for free. 2. Type a query into Search query, for example cat:cs.CL AND abs:retrieval. 3. Pick a Sort by. Use submittedDate when you care about what is new. 4. Set Max papers. Start around 20 while you tune the query, then click Start. 5. Download the dataset as JSON, CSV or Excel, or read it from the Apify API.

💰 How much does it cost?

$2.00 per 1,000 papers, which is $0.002 each. Flat on every Apify plan, no volume tiers.

You pay per paper row delivered. Duplicates across pages are dropped before they reach you, diagnostic rows are not charged, and a query that matches nothing costs you nothing.

💡 What people use it for

• Watching a category like cat:cs.CL on a schedule and pulling everything new since yesterday.
• Building a literature list on a topic, with every abstract already in the row to read or

summarise.

• Following one author with au: to see what they have put up and when it was last revised.
• Collecting PDF links for a set of papers to fetch and process in your own pipeline.

🚧 What it does not do

• No full text. Abstracts and links. The PDF is yours to fetch from pdfUrl.
• No citation counts, no references, no related-paper graph. arXiv does not publish them.
• No peer-review status. A paper on arXiv may be a preprint, a submitted manuscript or a

published article, and nothing on the row tells you which.

• doi is usually null. Most preprints have none, and arXiv does not go looking.
• No date-range filter. Sort by date and trim the rows yourself.
• Sort order is always newest or most relevant first. Ascending is not offered.
• About 30,000 results per query, maximum. That is arXiv's depth limit, not a setting here.

Split a broad query into narrower ones rather than raising the number.

• The run finishes even when every request failed, so read ok rather than the run status.

🧭 Which research scraper do you need?

❓ Questions people ask

Do I need an arXiv account or an API key? No. arXiv's search is open to anyone.

Why did my search return nothing? Usually the syntax. A bare phrase with no prefix is not a valid arXiv query; start it with all: and try again.

Can I get the paper itself? Not from this actor. pdfUrl gives you a direct link to fetch.

Which date should I sort on? submittedDate for when work first appeared, lastUpdatedDate for what was revised recently. They differ a lot on papers that keep being updated.

Can I schedule it? Yes. A daily cat: query sorted by submittedDate, deduped on arxivId, keeps a category feed current.

Is scraping arXiv legal? arXiv publishes metadata for public reuse and asks callers to be polite about request rate, which this actor is. Individual papers carry their own licences, so check those before republishing text. Apify's write-up on scraping and the law is a good starting point, and we are not lawyers.

🆘 If something breaks

Open the Issues tab on the actor page. Send the run ID and the exact query string you used. The errorCode and hint on the diagnostic row usually name the problem on their own.