Emerging Topic Scout

A real-time monitoring system for identifying "incubation period" research hotspots in biological and medical sciences before they are defined by mainstream journals.

Overview

This skill continuously monitors:

• - bioRxiv: Biology preprints via RSS/API ⚠️ Currently blocked by Cloudflare
• medRxiv: Medicine preprints via RSS/API ⚠️ Currently blocked by Cloudflare
• arXiv: Quantitative Biology preprints via RSS ✅ Recommended alternative
• Academic discussions: Social media and forum mentions

It uses trend analysis algorithms to detect sudden spikes in topic frequency, cross-platform mentions, and emerging keyword clusters.

⚠️ Network Access Notice

bioRxiv and medRxiv are currently protected by Cloudflare JavaScript Challenge, which prevents programmatic RSS access. As a workaround, this skill now supports arXiv q-bio (Quantitative Biology) as an alternative data source.

Recommended usage:
CODEBLOCK0

Installation

CODEBLOCK1

Usage

Basic Scan (Recommended: Use arXiv)

CODEBLOCK2

Legacy bioRxiv/medRxiv (May not work due to Cloudflare)

CODEBLOCK3

Advanced Configuration (arXiv Recommended)

CODEBLOCK4

Legacy Configuration (bioRxiv/medRxiv - May not work)

CODEBLOCK5json
{
"scan_date": "2026-02-06T05:57:00Z",
"sources": ["biorxiv", "medrxiv"],
"hot_topics": [
{
"topic": "gene editing therapy",
"keywords": ["CRISPR", "base editing", "prime editing"],
"trending_score": 0.89,
"velocity": "rapid",
"preprint_count": 34,
"crossplatformmentions": 127,
"related_papers": [
{
"title": "New CRISPR variant shows promise",
"authors": ["Smith J.", "Lee K."],
"doi": "10.1101/2026.01.15.xxxxx",
"source": "biorxiv",
"published": "2026-01-15",
"abstract_summary": "..."
}
],
"emerging_since": "2026-01-20"
}
],
"summary": {
"totalpapersanalyzed": 1247,
"newtopicsdetected": 8,
"highpriorityalerts": 2
}
}

### Markdown Output

markdown

Emerging Topics Report - 2026-02-06

🔥 High Priority Topics

1. Gene Editing Therapy (Score: 0.89)

• - Keywords: CRISPR, base editing, prime editing
• Growth Rate: Rapid (+145% vs last week)
• Preprints: 34 papers
• Cross-platform mentions: 127

Key Papers

1. 1. "New CRISPR variant shows promise" - Smith J. et al.

- DOI: 10.1101/2026.01.15.xxxxx - Source: bioRxiv

## Configuration File

Create `config.yaml` for persistent settings:

yaml sources: arxiv: enabled: true rss_url: "https://export.arxiv.org/rss/q-bio" description: "arXiv Quantitative Biology - Recommended (no Cloudflare)" biorxiv: enabled: false # Disabled due to Cloudflare protection rss_url: "https://www.biorxiv.org/rss/recent.rss" api_endpoint: "https://api.biorxiv.org/details/" note: "Currently blocked by Cloudflare JavaScript Challenge" medrxiv: enabled: false # Disabled due to Cloudflare protection rss_url: "https://www.medrxiv.org/rss/recent.rss" api_endpoint: "https://api.medrxiv.org/details/" note: "Currently blocked by Cloudflare JavaScript Challenge"

trending:
minpapersthreshold: 5
velocitywindowdays: 3
novelty_weight: 0.4
momentum_weight: 0.6

keywords:
auto_detect: true
custom_trackers:
- "artificial intelligence"
- "machine learning"
- "single cell"
- "spatial transcriptomics"

output:
default_format: markdown
save_history: true
history_path: "./data/history.json"

notifications:
enabled: false
highscorethreshold: 0.8

## Trending Score Algorithm

The trending score (0-1) is calculated using:

Score = (Novelty × 0.4) + (Momentum × 0.4) + (CrossRef × 0.2)

Where:

• - Novelty: Inverse frequency of topic in historical data
• Momentum: Rate of increase in mentions over velocity window
• CrossRef: Mentions across multiple platforms

## API Endpoints

### bioRxiv API
- Base: `https://api.biorxiv.org/`
- Details: `/details/[server]/[DOI]/[format]`
- Publication: `/pub/[DOI]/[format]`

### medRxiv API
- Same structure as bioRxiv

## Data Storage

Historical data is stored in `data/history.json` for:
- Trend comparison
- Velocity calculation
- Duplicate detection

## Examples

### Example 1: Quick Daily Scan (arXiv - Recommended)

bash
python scripts/main.py --sources arxiv --days 1 --output markdown

### Example 2: Daily Scan with bioRxiv (May not work)

bash
python scripts/main.py --sources biorxiv --days 1 --output markdown

Note: May return 0 results due to Cloudflare protection

Example 2: Weekly Deep Analysis

CODEBLOCK11

Example 3: Track Specific Research Area

CODEBLOCK12

Known Issues

bioRxiv/medRxiv Cloudflare Protection

Status: ❌ Blocked Issue: bioRxiv and medRxiv RSS feeds are protected by Cloudflare JavaScript Challenge, which prevents programmatic access. The site returns an HTML page requiring JavaScript execution and cookie validation.

Attempted Solutions:

1. 1. ✅ Added browser User-Agent headers → Failed (Cloudflare detects bot)
2. ✅ Added complete browser headers (Accept, Accept-Language, etc.) → Failed
3. ❌ Browser automation (Selenium/Playwright) → Not implemented (complex, heavy dependency)

Workaround: ✅ Use arXiv instead

• - arXiv q-bio (Quantitative Biology) RSS is accessible without protection
• Contains computational biology, bioinformatics, and quantitative biology papers
• Successfully tested: 35+ papers fetched in 30-day window

Usage:
CODEBLOCK13

Troubleshooting

Rate Limiting

If you encounter rate limits, increase the --delay parameter (default: 1s between requests).

Missing Papers (0 results from bioRxiv/medRxiv)

This is expected due to Cloudflare protection. Use --sources arxiv instead.

RSS Feed Access Denied

Some institutional firewalls may block preprint servers. Ensure you can access:

• - ✅ https://export.arxiv.org/rss/q-bio (should work)
• ❌ https://www.biorxiv.org/rss/recent.rss (Cloudflare blocked)

Low Trending Scores

For niche topics, lower --min-score threshold or increase --days for more data.

References

See references/README.md for:

• - API documentation links
• Research papers on trend detection
• Related tools and resources

License

MIT License - Part of OpenClaw Skills Collection

Risk Assessment

Risk Indicator	Assessment	Level
Code Execution	Python scripts with tools	High
Network Access

External API calls | High | | File System Access | Read/write data | Medium | | Instruction Tampering | Standard prompt guidelines | Low | | Data Exposure | Data handled securely | Medium |

Security Checklist

• - [ ] No hardcoded credentials or API keys
• [ ] No unauthorized file system access (../)
• [ ] Output does not expose sensitive information
• [ ] Prompt injection protections in place
• [ ] API requests use HTTPS only
• [ ] Input validated against allowed patterns
• [ ] API timeout and retry mechanisms implemented
• [ ] Output directory restricted to workspace
• [ ] Script execution in sandboxed environment
• [ ] Error messages sanitized (no internal paths exposed)
• [ ] Dependencies audited
• [ ] No exposure of internal service architecture

Prerequisites

CODEBLOCK14

Evaluation Criteria

Success Metrics

• - [ ] Successfully executes main functionality
• [ ] Output meets quality standards
• [ ] Handles edge cases gracefully
• [ ] Performance is acceptable

Test Cases

1. 1. Basic Functionality: Standard input → Expected output
2. Edge Case: Invalid input → Graceful error handling
3. Performance: Large dataset → Acceptable processing time

Lifecycle Status

• - Current Stage: Draft
• Next Review Date: 2026-03-06
• Known Issues:

- ⚠️ bioRxiv/medRxiv blocked by Cloudflare (use arXiv as workaround) - Network access limitations for some RSS feeds

• - Planned Improvements:

- Investigate bioRxiv/medRxiv API alternatives - Consider browser automation for Cloudflare bypass - Add more arXiv categories (q-bio subcategories) - Performance optimization