Crossref Scholarly Works Scraper

Crossref Scraper: search scholarly works by keyword and get the DOI back

Type a research phrase and get back the works Crossref has a DOI for: journal articles, preprints, book chapters, datasets, dissertations. Every row carries the DOI, the title, the authors, the journal, the publisher, the publication date and the citation count.

The honest part first. Crossref holds what publishers deposited with it, and plenty of them deposit the bare minimum. Abstracts, subject terms and ISSNs are missing on a lot of older records, and the citation count reads 0 both when a work genuinely has none and when nobody has told Crossref.

🔍 What Crossref Scraper does

It runs your keywords against Crossref's search, which reads across titles, authors, abstracts and the rest of the deposited metadata, then pages through the matches with a cursor so a large request keeps going past the first hundred.

You can narrow it to one work type or to particular journals by ISSN, set a published-on-or-after floor, and choose whether the results come back by relevance, by citation count or newest first. Works with no DOI are skipped, and a DOI that appears twice across pages is only delivered once.

Nothing is written to your dataset until the whole set has been collected, so you get either the full result set you asked for or a diagnostic row explaining why not.

📥 What you give it

{
  "query": "CRISPR gene editing",
  "filterType": "journal-article",
  "fromDate": "2020-01-01",
  "sort": "is-referenced-by-count",
  "maxItems": 500
}

fromDate has to be a real date shaped YYYY-MM-DD, and an ISSN has to be 8 characters like 0950-382X. Anything else stops the run with a BAD_INPUT row before a single search is made.

📤 What you get back

A real row from a recent run:

{
  "ok": true,
  "doi": "10.1067/mva.1993.50616",
  "title": "Light reflection rheography: A simple noninvasive screening test for deep vein thrombosis",
  "authors": ["Thomas W. Wakefield", "Subodh W. Arora", "David J. K. Lam", "..."],
  "journal": "Journal of Vascular Surgery",
  "publisher": "Elsevier BV",
  "type": "journal-article",
  "publishedDate": "1993-11",
  "citations": 0,
  "subjects": [],
  "issn": ["0741-5214"],
  "abstract": null,
  "url": "https://doi.org/10.1067/mva.1993.50616"
}

The author list on that row is cut short here for length. A real row carries all six names.

🧾 Reading the output

Two kinds of row land in your dataset, and ok tells them apart.

The overview table in the Apify console shows the work columns only, so a diagnostic row looks blank there. Switch to the JSON or All fields view to read it.

▶️ How to run it

1. Open Crossref Scraper and click Try for free. 2. Type your keywords into Search query. 3. Set Max works. Start around 50 to see the row shape. 4. Pick a Work type filter and a Sort order if the defaults do not suit you, then click Start. 5. Download the dataset as JSON, CSV or Excel, or read it from the Apify API.

💰 How much does it cost?

$1.00 per 1,000 works. Flat on every Apify plan, no volume tiers.

You pay per work delivered. Duplicate DOIs and works without a DOI are dropped before they are counted, diagnostic rows are not charged, and a search that matches nothing is not charged.

💡 What people use it for

• Building a DOI list for a literature review, then fetching the papers themselves elsewhere.
• Sorting a field by citation count to see which work everything else is built on.
• Watching a topic on a schedule with sort on published, so each run surfaces what is new.
• Checking which publisher holds a set of papers before writing to them about access.
• Feeding titles and abstracts into a model that has to summarise a field it was not trained on.

🚧 What it does not do

• No lookup by DOI or by author. Keywords and journal ISSNs are the ways in.
• No end date. fromDate sets a floor and nothing sets a ceiling.
• Newest first stops at 10,000 works, and a few records carry publication dates years ahead,

which sort to the top. In one test the first rows were dated 2115 and 2106. Deep in that order, works sharing a date can repeat across pages; repeats are dropped and not charged, so a full run ends a little short (9,872 of 10,000 in that test).

• Abstracts are mostly missing. Crossref only has one if the publisher deposited it, and many

never do.

• citations cannot tell you zero from unknown. Both read 0.
• No full text, and no PDF. You get the DOI and the link, not the paper.
• Paging stops on a short page. If Crossref returns fewer than 100 items in one page, the run

treats that as the end even when its own total says otherwise.

• All or nothing. A failure partway through discards what had been collected, so you get a

diagnostic row rather than a partial set.

• Crossref is publisher-deposited metadata, not a curated index. Quality varies by publisher and

by decade.

🧭 Which research scraper do you need?

❓ Questions people ask

Do I need an API key? No. Crossref publishes this openly and the actor talks to it directly.

Why are so many abstracts null? Crossref stores what the publisher sent. Most publishers never deposit an abstract, so the field is empty far more often than it is filled.

Can I look a paper up by its DOI? Not here. This searches by keyword and by journal ISSN.

Can I list what one journal published? Yes. Put its ISSN in issn and leave the query empty. A big journal holds more than one run delivers, so sort newest first or by citations to get the end you care about.

Why did I get fewer works than maxItems? Either Crossref has fewer matches, or a page came back short and paging stopped there. You are charged for what arrived.

Can I get the PDF? No. You get the DOI and the publisher's link, and access depends on the publisher.

Is this legal? Crossref publishes this metadata openly for exactly this kind of use. Apify's write-up on scraping and the law is a good starting point, and we are not lawyers.

🆘 If something breaks

Open the Issues tab on the actor page. Send the run ID and the query you used. The errorCode on the diagnostic row usually names the problem on its own.