Crawlee Skill

Crawlee is a production-grade web scraping and browser automation library for JavaScript/TypeScript (Node.js 16+)
and Python (3.10+). It handles anti-blocking, proxies, session management, storage, and concurrency out of the box.

Docs: https://crawlee.dev/js/docs | https://crawlee.dev/python/docs
GitHub: https://github.com/apify/crawlee

1. Choose Your Crawler

JavaScript / TypeScript

Rule of thumb: Start with CheerioCrawler. Upgrade to PlaywrightCrawler only when JS rendering is required.

Python

2. Installation

JavaScript

Add to package.json:
CODEBLOCK1

Python

pip install crawlee

# With BeautifulSoup:
pip install 'crawlee[beautifulsoup]'

# With Playwright:
pip install 'crawlee[playwright]'
playwright install

3. Core Concepts

The Two Questions Every Crawler Answers

1. 1. Where to go? → Request objects in a INLINECODE15
2. What to do there? → requestHandler function (JS) / decorated handler (Python)

Key Classes (JS)

• - Request — A single URL + metadata to crawl
• INLINECODE18 — Dynamic, deduplicated queue of URLs
• INLINECODE19 — Append-only structured result storage (like a table)
• INLINECODE20 — Blob storage for screenshots, PDFs, state
• INLINECODE21 — Manages proxy rotation
• INLINECODE22 — Manages browser sessions + cookies

4. Quick Start Examples

JavaScript — CheerioCrawler (Recommended Start)

JavaScript — PlaywrightCrawler

Python — BeautifulSoupCrawler

Python — PlaywrightCrawler

import asyncio
from crawlee.crawlers import PlaywrightCrawler, PlaywrightCrawlingContext

async def main() -> None:
    crawler = PlaywrightCrawler(headless=True, browser_type='chromium')

    @crawler.router.default_handler
    async def handler(context: PlaywrightCrawlingContext) -> None:
        title = await context.page.title()
        await context.push_data({'url': context.request.url, 'title': title})
        await context.enqueue_links()

    await crawler.run(['https://example.com'])

if __name__ == '__main__':
    asyncio.run(main())

5. Routing — Handling Multiple Page Types

Use labels + router to handle different kinds of pages (list pages, detail pages, etc.).

JavaScript

CODEBLOCK8

Python

from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext

crawler = BeautifulSoupCrawler()

@crawler.router.handler('CATEGORY')
async def category_handler(context: BeautifulSoupCrawlingContext) -> None:
    await context.enqueue_links(selector='a.product', label='DETAIL')

@crawler.router.default_handler
async def detail_handler(context: BeautifulSoupCrawlingContext) -> None:
    title = context.soup.title.string
    await context.push_data({'url': context.request.url, 'title': title})

6. Enqueuing Links

JavaScript — enqueueLinks()

Python

await context.enqueue_links()
await context.enqueue_links(selector='a.product', label='DETAIL')
await context.enqueue_links(include=[re.compile(r'/products/\d+')])

7. Storage

Dataset (structured results)

CODEBLOCK13

Data is saved to ./storage/datasets/default/*.json by default.

KeyValueStore (blobs, screenshots, state)

CODEBLOCK15

Storage location

Override with env var: CRAWLEE_STORAGE_DIR=/path/to/storage

8. Proxy Management

CODEBLOCK17

CODEBLOCK18

CODEBLOCK19

9. Session Management

Sessions tie together cookies, proxy IPs, and headers to simulate a consistent user identity.

CODEBLOCK20

CODEBLOCK21

10. Avoiding Blocks

CODEBLOCK22

Anti-blocking checklist:

• - ✅ Use CheerioCrawler — it uses got-scraping which mimics real browser HTTP
• ✅ Enable useSessionPool: true with a INLINECODE29
• ✅ Use tiered proxies for automatic failover
• ✅ Set maxRequestsPerMinute to avoid rate limits
• ✅ For browser crawlers — fingerprints are rotated automatically
• ✅ Use INLINECODE31
• ✅ Retire sessions on blocks: INLINECODE32

11. Concurrency & Scaling

CODEBLOCK23

CODEBLOCK24

Scaling notes:

• - Crawlee auto-scales concurrency based on CPU/memory
• Don't set minConcurrency high — it can crash under load
• INLINECODE34 is smoother than raw concurrency throttling

12. Configuration & Environment Variables

CODEBLOCK25

13. Docker Deployment

CODEBLOCK26

For Cheerio (smaller image):

FROM apify/actor-node:20

14. Common Patterns

Pagination

Downloading Files

Taking Screenshots

Shared State Across Handlers

Error Handling & Retries

CODEBLOCK33

Sitemap Crawling

Run as Web Server

import { CheerioCrawler } from 'crawlee';
import { createServer } from 'http';

const server = createServer(async (req, res) => {
  const url = new URL(req.url, 'http://localhost').searchParams.get('url');
  const crawler = new CheerioCrawler({
    maxRequestsPerCrawl: 1,
    async requestHandler({ $ }) {
      res.end(JSON.stringify({ title: $('title').text() }));
    },
  });
  await crawler.run([url]);
});
server.listen(3000);

15. TypeScript Support

CODEBLOCK36

16. Cloud Deployment (Apify Platform)

CODEBLOCK37

Deploy with: apify push

17. Debugging Tips

CODEBLOCK38

18. Reference Files

For advanced topics, see:

• - references/js-api.md — Full JS API quick reference
• INLINECODE47 — Full Python API quick reference

Both language docs: https://crawlee.dev