Hacker News Scraper

Hacker News Scraper: search stories, Show HN, Ask HN and comments, no API key

Type a query, pick what kind of item you want, and the matches come back as rows: points, author, comment count, the story's own link and the HN thread link. Stories, Show HN, Ask HN, comments, or whatever is sitting on the front page right now.

One thing to know before the first run. The search box on the input form starts out filled in with openai, and the query is applied to the front page as well. Leave it there with Front page selected and you get the few front-page items that mention OpenAI, not the front page.

🔎 What Hacker News Scraper does

It searches the same index that the search box at the foot of every Hacker News page uses. That index is public and needs no key, so there is nothing to sign up for and no quota to watch.

Pick the item type and you get stories, Show HN posts, Ask HN posts, comments, or the current front page. Leave the query empty and you get the newest items of that type instead of a keyword search. Sort by relevance or by date. Set a minimum score, or a minimum number of comments, to drop the quiet stories before they reach your dataset.

It pages through results 50 at a time until it has what you asked for or the index runs out. Items that appear on two pages are dropped once, so the same story cannot land twice in one run.

📥 What you give it

{
  "query": "rust async",
  "tags": "story",
  "sortBy": "date",
  "minPoints": 50,
  "maxItems": 200
}

📤 What you get back

A real row from a recent run:

{
  "ok": true,
  "objectId": "38309611",
  "type": "story",
  "title": "OpenAI's board has fired Sam Altman",
  "url": "https://openai.com/blog/openai-announces-leadership-transition",
  "author": "davidbarker",
  "points": 5710,
  "numComments": 2530,
  "createdAt": "2023-11-17T20:28:50Z",
  "text": "",
  "hnUrl": "https://news.ycombinator.com/item?id=38309611"
}

🧾 Reading the output

Data rows carry ok: true. When something goes wrong, or a search matches nothing, you get one row with ok: false and an errorCode instead of an empty dataset, and that row is not charged.

One more thing about the Console's overview table: it shows title, author, points, comments and the two links, and it has no column for text. So a comments run looks odd there, every row showing its parent story's title with no comment body. Switch to all fields, or export the JSON, and the text is there.

▶️ How to run it

1. Open Hacker News Scraper and click Try for free. 2. Type your keywords into Search query, or clear the box to get the newest items instead. 3. Pick an Item type. Stories is the default, comments is the one you want for mention tracking. 4. Set Max items. Start around 20 to see the shape of the output, then click Start. 5. Download the dataset as JSON, CSV or Excel, or read it from the Apify API.

💰 How much does it cost?

$1.00 per 1,000 items. Flat on every Apify plan, no volume tiers.

You pay per row delivered, so maxItems is your budget dial and the default of 50 is deliberately small. Items dropped by minPoints or minComments, duplicates removed inside a run, and diagnostic rows are not charged. A query that matches nothing writes one diagnostic row and costs you nothing.

💡 What people use it for

• Mention tracking. Put your product name in the query, set the type to comments, sort by date and

run it every few hours. Threads move fast and hearing about yours from a customer is worse.

• A feed of new launches: Show HN, sorted by date, no query needed.
• The monthly "Who is hiring" lists. Those are comments under a post rather than the post itself, so

search hiring with the type set to comments.

• A front-page digest on a schedule, with the query box cleared.
• Checking how a topic has been received over the years, sorted by points.

🚧 What it does not do

• 1,000 rows per run. Split a bigger job across several queries or date ranges.
• No full threads. You get the comments that match your search, not every reply under a story.
• No user profiles, no karma totals, no submission history for a person.
• The front page is filtered by your query too. For a plain digest, clear the query.
• A minimum score does nothing useful on comments. Comment rows carry no score, so a run with

both usually comes back with nothing at all. A minimum number of comments is refused on comments outright, for the same reason.

• Scores are a snapshot. A story read an hour after it was posted has the points it had then.
• No comment counts on comments. Those fields are for stories.
• Relevance ordering is the index's own, and it is not something this actor can change.

🧭 Which news scraper do you need?

❓ Questions people ask

Do I need a Hacker News account or an API key? Neither. Nothing to apply for.

What happens when my search matches nothing? You get one row with ok: false and errorCode: NO_RESULTS, and you are not charged for it.

Can I watch for mentions on a schedule? Yes. Schedule it, sort by date, and dedupe on objectId so you only act on what is new since the last run.

Why did my front-page run come back nearly empty? The query is applied to the front page as well, and the form starts with openai in the box. Clear it.

Can I pull a whole comment thread? No. Search returns matching comments, not the tree under a story. The hnUrl on each row opens the thread if you want to read it.

Is this legal? These are public posts on a public site, and the search index behind them is public too. Apify's write-up on scraping and the law is a good starting point, and we are not lawyers.

🆘 If something breaks

Open the Issues tab on the actor page. Send the query you used and the run ID. The errorCode on the diagnostic row usually names the problem on its own.