scraper

# docs-scraper CLI tool that scrapes documents from various sources into local PDF files using browser automation. ## Installation ```bash npm install -g docs-scraper ``` ## Quick start Scrape any document URL to PDF: ```bash docs-scraper scrape https://example.com/document ``` Returns local path: `~/.docs-scraper/output/1706123456-abc123.pdf` ## Basic scraping **Scrape with daemon** (recommended, keeps browser warm): ```bash docs-scraper scrape <url> ``` **Scrape with named profile** (for authenticated sites): ```bash docs-scraper scrape <url> -p <profile-name> ``` **Scrape with pre-filled data** (e.g., email for DocSend): ```bash docs-scraper scrape <url> -D email=user@example.com ``` **Direct mode** (single-shot, no daemon): ```bash docs-scraper scrape <url> --no-daemon ``` ## Authentication workflow When a document requires authentication (login, email verification, passcode): 1. Initial scrape returns a job ID: ```bash docs-scraper scrape https://docsend.com/view/xxx # Output: Scrape blocked # Job ID: abc123 ``` 2. Retry with data: ```bash docs-scraper update abc123 -D email=user@example.com # or with password docs-scraper update abc123 -D email=user@example.com -D password=1234 ``` ## Profile management Profiles store session cookies for authenticated sites. ```bash docs-scraper profiles list # List saved profiles docs-scraper profiles clear # Clear all profiles docs-scraper scrape <url> -p myprofile # Use a profile ``` ## Daemon management The daemon keeps browser instances warm for faster scraping. ```bash docs-scraper daemon status # Check status docs-scraper daemon start # Start manually docs-scraper daemon stop # Stop daemon ``` Note: Daemon auto-starts when running scrape commands. ## Cleanup PDFs are stored in `~/.docs-scraper/output/`. The daemon automatically cleans up files older than 1 hour. Manual cleanup: ```bash docs-scraper cleanup # Delete all PDFs docs-scraper cleanup --older-than 1h # Delete PDFs older than 1 hour ``` ## Job management ```bash docs-scraper jobs list # List blocked jobs awaiting auth ``` ## Supported sources - **Direct PDF links** - Downloads PDF directly - **Notion pages** - Exports Notion page to PDF - **DocSend documents** - Handles DocSend viewer - **LLM fallback** - Uses Claude API for any other webpage --- ## Scraper Reference Each scraper accepts specific `-D` data fields. Use the appropriate fields based on the URL type. ### DirectPdfScraper **Handles:** URLs ending in `.pdf` **Data fields:** None (downloads directly) **Example:** ```bash docs-scraper scrape https://example.com/document.pdf ``` --- ### DocsendScraper **Handles:** `docsend.com/view/*`, `docsend.com/v/*`, and subdomains (e.g., `org-a.docsend.com`) **URL patterns:** - Documents: `https://docsend.com/view/{id}` or `https://docsend.com/v/{id}` - Folders: `https://docsend.com/view/s/{id}` - Subdomains: `https://{subdomain}.docsend.com/view/{id}` **Data fields:** | Field | Type | Description | |-------|------|-------------| | `email` | email | Email address for document access | | `password` | password | Passcode/password for protected documents | | `name` | text | Your name (required for NDA-gated documents) | **Examples:** ```bash # Pre-fill email for DocSend docs-scraper scrape https://docsend.com/view/abc123 -D email=user@example.com # With password protection docs-scraper scrape https://docsend.com/view/abc123 -D email=user@example.com -D password=secret123 # With NDA name requirement docs-scraper scrape https://docsend.com/view/abc123 -D email=user@example.com -D name="John Doe" # Retry blocked job docs-scraper update abc123 -D email=user@example.com -D password=secret123 ``` **Notes:** - DocSend may require any combination of email, password, and name - Folders are scraped as a table of contents PDF with document links - The scraper auto-checks NDA checkboxes when name is provided --- ### NotionScraper **Handles:** `notion.so/*`, `*.notion.site/*` **Data fields:** | Field | Type | Description | |-------|------|-------------| | `email` | email | Notion account email | | `password` | password | Notion account password | **Examples:** ```bash # Public page (no auth needed) docs-scraper scrape https://notion.so/Public-Page-abc123 # Private page with login docs-scraper scrape https://notion.so/Private-Page-abc123 \ -D email=user@example.com -D password=mypassword # Custom domain docs-scraper scrape https://docs.company.notion.site/Page-abc123 ``` **Notes:** - Public Notion pages don't require authentication - Toggle blocks are automatically expanded before PDF generation - Uses session profiles to persist login across scrapes --- ### LlmFallbackScraper **Handles:** Any URL not matched by other scrapers (automatic fallback) **Data fields:** Dynamic - determined by Claude analyzing the page The LLM scraper uses Claude to analyze the page HTML and detect: - Login forms (extracts field names dynamically) - Cookie banners (auto-dismisses) - Expandable content (auto-expands) - CAPTCHAs (reports as blocked) - Paywalls (reports as blocked) **Common dynamic fields:** | Field | Type | Description | |-------|------|-------------| | `email` | email | Login email (if detected) | | `password` | password | Login password (if detected) | | `username` | text | Username (if login uses username) | **Examples:** ```bash # Generic webpage (no auth) docs-scraper scrape https://example.com/article # Webpage requiring login docs-scraper scrape https://members.example.com/article \ -D email=user@example.com -D password=secret # When blocked, check the job for required fields docs-scraper jobs list # Then retry with the fields the scraper detected docs-scraper update abc123 -D username=myuser -D password=secret ``` **Notes:** - Requires `ANTHROPIC_API_KEY` environment variable - Field names are extracted from the page's actual form fields - Limited to 2 login attempts before failing - CAPTCHAs require manual intervention --- ## Data field summary | Scraper | email | password | name | Other | |---------|-------|----------|------|-------| | DirectPdf | - | - | - | - | | DocSend | ✓ | ✓ | ✓ | - | | Notion | ✓ | ✓ | - | - | | LLM Fallback | ✓* | ✓* | - | Dynamic* | *Fields detected dynamically from page analysis ## Environment setup (optional) Only needed for LLM fallback scraper: ```bash export ANTHROPIC_API_KEY=your_key ``` Optional browser settings: ```bash export BROWSER_HEADLESS=true # Set false for debugging ``` ## Common patterns **Archive a Notion page:** ```bash docs-scraper scrape https://notion.so/My-Page-abc123 ``` **Download protected DocSend:** ```bash docs-scraper scrape https://docsend.com/view/xxx # If blocked: docs-scraper update <job-id> -D email=user@example.com -D password=1234 ``` **Batch scraping with profiles:** ```bash docs-scraper scrape https://site.com/doc1 -p mysite docs-scraper scrape https://site.com/doc2 -p mysite ``` ## Output **Success**: Local file path (e.g., `~/.docs-scraper/output/1706123456-abc123.pdf`) **Blocked**: Job ID + required credential types ## Troubleshooting - **Timeout**: `docs-scraper daemon stop && docs-scraper daemon start` - **Auth fails**: `docs-scraper jobs list` to check pending jobs - **Disk full**: `docs-scraper cleanup` to remove old PDFs