🛠 Shipping Signal: Building a Builder-First Hacker News Digest

Every week I read Hacker News like a hungry engineer with too many tabs and not enough patience. What I needed wasn’t more news—it was better-filtered noise. Something that knew what actually mattered to builders. Not funding rounds, not BigCo PR. Just raw, high-signal links about the stuff we love: infrastructure, dev tools, Python oddities, agent frameworks, CLI hacks.

So I built it.

python research.py --keyword ai python cli --limit 50

A one-command Hacker News digest, filtered for builders, written in Markdown, styled with care, and optionally—crafted by AI using CrewAI.

👉 Source on GitHub: crafty-arl/CrewAIHackerNewsBot

🧱 Stack

• Python – for scraping and orchestration

• BeautifulSoup – to wrangle the web

• CrewAI + OpenAI – to editorialize stories, builder-style

• .env setup – because leaking API keys is 2019 energy

• HN Firebase API – because scraping the front page is old hat

✂️ Filtering for Signal

No clickbait. Just signal.

DEFAULT_KEYWORDS = [ "infrastructure", "stack", "python", "open source", "devops", "cli", "tool", "github", "cloud", "langchain", "ai", "agent" ]

The script grabs the latest 500 story IDs from Hacker News, checks titles against these keywords, and filters by recency (this month only). Then it scrapes the article’s body text (first 2,000 chars—token-safe) and hands it off to an AI writing agent with a very specific voice:

🧠 "Builder-First Tech Writer" who respects technical depth but doesn’t flex. Thinks in markdown. Writes with care, not clout."

✍️ Human-Aware, Builder-Toned Output

If --simple is passed, it dumps a link-only digest like:

# /build Digest – 2025-04-24 ## Hacker News Highlights (Filtered by: devops, ai, python) - [Why Your Infra Should Be Boring](https://news.ycombinator.com/item?id=...) - [LangChain Isn’t Magic: A Cautionary Tale](https://news.ycombinator.com/item?id=...) - [Python CLI Tools That Actually Work](https://news.ycombinator.com/item?id=...)

But if you run it without --simple, it runs a full CrewAI pipeline to write a proper blog post with intro, summaries, and builder-aware commentary. Because sometimes, Markdown deserves more than just a list.

💡 Lessons in the Build

• Respect the throttle: Scraping ≠ scraping fast. I sleep 1s between requests.

• Keyword tuning is everything: “cli” is better than “command line tool” for Hacker News.

• Markdown is a UX choice: Developers don’t want newsletter fluff—they want ctrl+f-able context.

✅ What’s Next

• Add emoji/category tags per link

• Option to sort by comment count

• Ship a --weekly cron mode

• Feed it into a static blog with git-based history

🧵 Call to Builders

This isn’t just a tool. It’s a stance. A stance against firehose feeds and AI-generated noise. A stance for dev-first curation, for people who build side projects at midnight and ship posts that actually help others.

If you're one of us, fork it. Remix it. Run your own digest.

→ GitHub: crafty-arl/CrewAIHackerNewsBot

Want to help shape what this digest highlights? Drop your keywords or RSS sources in a PR. Or just run the script and own your own corner of the web.

👊 Built for builders, by someone tired of pretending every launch is a revolution.