Multi-Source Feed

AI-curated daily tech brief aggregated from customizable sources (X, HN, GitHub Trending, RSS blogs, Reddit, Product Hunt, Tavily, and more). Deduplicates, filters by your interests, and delivers a structured memo.

Setup

When the user asks to set up Multi-Source Feed, follow these steps in order. Execute each step automatically. If any step fails, print the manual command and continue.

Step 1: Clone & Install

CODEBLOCK0

If clone already exists, skip to pip install.

Step 2: API Keys

Some sources require API keys that the user must register themselves. Ask the user for:

1. 1. Tavily (free): powers web search to catch trending topics not covered by RSS feeds. Sign up at https://tavily.com
2. Product Hunt (free): required for the Product Hunt GraphQL API. Get a token at https://api.producthunt.com/v2/docs

Once the user provides both keys, write them to ~/multi-source-feed/.env:

CODEBLOCK1

Step 3: X/Twitter Login

Tell the user:

"To save your X/Twitter session, you need to:

1. 1. Open Chrome with remote debugging enabled by running:

open -a 'Google Chrome' --args --remote-debugging-port=9222

1. 2. Log in to X/Twitter in that Chrome window
2. Once logged in, I'll run a script that connects to that browser and saves your session cookies."

After the user confirms they are logged in to X in Chrome, run:

CODEBLOCK2

This script connects to the already-open Chrome instance via CDP (Chrome DevTools Protocol) on port 9222, extracts the session/cookies, and saves them to x_session.json in the project root. It does not open a new browser window — it requires Chrome to already be running with --remote-debugging-port=9222.

Step 4: Customize

This step directly affects the quality of the daily brief. Strongly encourage the user to customize before proceeding.

Ask the user:

"The default profile is a generic template. I strongly recommend customizing these files to match your interests — this directly determines the quality of your daily brief. What topics do you care about? What should be filtered out?"

Based on their response:

• - Edit config/user_profile.md — set their interests, non-interests, and Key Players to track
• Adjust config/sources.yaml if needed (enable/disable sources, add their own RSS feeds)
• Adjust config/preferences.md if they want different memo sections or format

If they insist on skipping, move on — but remind them they can customize later.

Step 5: Test Run

CODEBLOCK3

Show the user the output summary (number of sources, items fetched, any errors). If successful, show 5-10 sample titles from feed_slim.json.

Step 6: Schedule

The system runs in two phases. Phase 1 (scraping) must complete before Phase 2 (memo generation) starts.

Phase 1: Scrape (crontab) — Pure Python job that fetches all sources, deduplicates, and writes feed_slim.json. Set up a daily cron job:
CODEBLOCK4

Phase 2: Memo (OpenClaw cron) — LLM-powered job that generates the daily brief and sends it to the user. Must run ~20 min after Phase 1 to ensure scraping is complete.

Create an OpenClaw cron job that:

1. 1. Checks if feed_slim.json exists and is from today
2. Reads config/user_profile.md and INLINECODE12
3. Reads feed_slim.json (the scrape output)
4. Generates the daily brief following preferences.md format
5. Sends the brief to the user via their configured channel
6. Saves the brief to memo/YYYY-MM-DD.md (used for cross-day dedup)

Tell the user:

"Setup complete! Your daily brief will be generated every morning. You'll receive it through your configured messaging channel."

Manual Setup Fallback

If automated setup fails, provide the user with these commands to run manually:

CODEBLOCK5

Customization

All user-customizable files are in config/:

Adding a new RSS source

Add 4 lines to config/sources.yaml:
CODEBLOCK6

X-Push (Optional)

If the user wants real-time X/Twitter highlights every 2 hours (in addition to the daily brief):

1. 1. Ask the user if they want to customize push/preferences.md (filtering rules and output format)
2. Create an OpenClaw cron job (every 2 hours) with this prompt:

CODEBLOCK7

The push module shares x_session.json and .venv with the main pipeline — no extra setup needed.