Stack Overflow / Stack Exchange Scraper

Stack Overflow Scraper: questions, tags, scores and the question body

Search Stack Overflow and five sibling Stack Exchange sites by keywords, by tags, or by both. Each row is one question: the title, the score, how many answers it has, the view count, its tag list, who asked it and their reputation, and the whole question body as plain text.

Questions first. Turn on includeAcceptedAnswer and each row also carries the accepted answer as plain text. The other answers and the comments are not in the output, so if you need whole threads, this is the wrong purchase however cheap it is.

🔍 What Stack Overflow Scraper does

It runs one search and pages through it until it has as many questions as you asked for.

query matches words in titles and bodies. tags filters, and it is an AND: put javascript,promise and you get only questions carrying both, not either. You can use tags on their own with no keywords, which is the cleanest way to pull the top questions in a tag.

sortBy decides the order, and it changes what a small run gives you. votes returns the long-standing classics. creation returns what was asked most recently. activity returns what people are still arguing about. relevance leans on the keyword match. On votes you can also set minScore to keep only questions at or above a score.

Six sites are available: Stack Overflow, Server Fault, Super User, Ask Ubuntu, MathOverflow and Software Engineering. One site per run.

With includeAcceptedAnswer on, each question also carries the answer its asker accepted: score, author and the body as plain text. Many questions never get one, and those say so with null.

📥 What you give it

{
  "query": "async await",
  "tags": "javascript,promise",
  "site": "stackoverflow",
  "sortBy": "votes",
  "maxItems": 50
}

Send a query, tags, or both. Leaving both empty is not refused, and what comes back then is whatever the site decides to hand over, charged like any other rows. It is the one input mistake here that costs money.

📤 What you get back

A real row from a recent run, with the body cut short here for length:

{
  "ok": true,
  "questionId": 37576685,
  "title": "Using async/await with a forEach loop",
  "url": "https://stackoverflow.com/questions/37576685/using-async-await-with-a-foreach-loop",
  "score": 3353,
  "answerCount": 35,
  "viewCount": 2481345,
  "isAnswered": true,
  "tags": ["javascript", "node.js", "promise", "async-await", "ecmascript-2017"],
  "ownerName": "Saad",
  "ownerReputation": 55029,
  "createdAt": "2016-06-01T18:55:58.000Z",
  "body": "Are there any issues with using async/await in a forEach loop? I'm trying to loop through an array of files and await on the contents of each file. ..."
}

🧾 Reading the output

Questions carry ok: true. Anything with ok: false carries an errorCode and is not charged.

A RATE_LIMITED row after some questions is not a failed run. It means the run delivered what it could and stopped rather than spinning. The same goes for an acceptedAnswerId with an empty acceptedAnswer: that answer could not be read in this run, usually because the day's allowance ran out. The run log says how many were read.

The Console's Overview table hides tags, isAnswered and body. Download the dataset as JSON, CSV or Excel, or switch the table to all fields.

▶️ How to run it

1. Open Stack Overflow Scraper and click Try for free. 2. Type into Search query, or fill Tags instead, or both. 3. Pick a Stack Exchange site if you want one of the other five. 4. Choose a Sort by order and set Max questions, then click Start. 5. Download the dataset as JSON, CSV, Excel or XML.

💰 How much does it cost?

$2.00 per 1,000 questions. Flat on every Apify plan, no volume tiers.

You pay per question row. Duplicates removed inside a run and every diagnostic row are not charged, and a search that matches nothing does not bill for results. Questions below minScore are never fetched, so they cost nothing either. The accepted answer adds nothing to the price.

💡 What people use it for

• Finding the questions a product's documentation should have answered, by searching its name and

reading what people are stuck on.

• Comparing tag volume between two libraries before committing to one.
• Building an evaluation set for a coding model out of plain-text question bodies.
• Watching a tag weekly with sortBy on creation to see what problems are new.

🚧 What it does not do

• The accepted answer only, and only when you ask for it. Other answers and comments are not

read. answerCount tells you how many answers there are.

• One site per run. Searching Stack Overflow and Ask Ubuntu together means two runs.
• No date range input. Sort by creation and filter on createdAt afterwards.
• Only six of the network's sites, the ones in the list above.
• Tags are AND, never OR. Three tags usually returns far less than you expect.
• There is a daily ceiling on how much anyone can pull without a key. When it is reached the run

stops and says so instead of failing. Accepted answers count against it too, about one extra read per hundred questions.

• Rows are a snapshot. Scores, views and answer counts move.

🧭 Which developer data scraper do you need?

❓ Questions people ask

Do I need a Stack Exchange API key? No.

Can I search tags without keywords? Yes, and it is the better way to pull a tag's top questions. Leave the query empty and fill tags.

Why did two tags return almost nothing? Because both have to be on the same question. Drop one.

Can I get the answers? The accepted one, yes: turn on includeAcceptedAnswer. The others are not read, so take url from the row and open the question.

Why does my title contain "? The title is passed through as the API sends it. Decode it on your side, or use body, which is already plain text.

Is this legal? Stack Exchange publishes this through a public API meant to be read, and the content is under a Creative Commons licence that asks for attribution. Apify's write-up on scraping and the law is a good starting point, and we are not lawyers.

🆘 If something breaks

Open the Issues tab on the actor page. Send the query, the tags and the run id. The errorCode on the diagnostic row usually names the problem by itself.