个人中心
今日签到
私信列表
消息中心
搜索全站
q_code

扫码关注官方微信
cell_code

扫码下载APP
返回顶部
技能集市 > AI智能 > scrapling
s

scrapling自适应网页抓取

Adaptive web scraping framework with anti-bot bypass and spider crawling.

作者: admin | 来源: ClawHub
下载
源自
ClawHub
版本
V 1.0.8
安全检测
已通过
2,678
下载量
免费
免费
3
收藏
概述
安装方式
版本历史

scrapling

Scrapling - 自适应网页抓取

为现代网页提供轻松的网页抓取体验。

致谢

核心库

  • - 仓库地址： https://github.com/D4Vinci/Scrapling
  • 作者： D4Vinci (Karim Shoair)
  • 许可证： BSD-3-Clause
  • 文档： https://scrapling.readthedocs.io

API逆向工程方法论

  • - GitHub： https://github.com/paoloanzn/free-solscan-api
  • X帖子： https://x.com/paoloanzn/status/2026361234032046319
  • 作者： @paoloanzn
  • 洞见： 网页抓取80%是逆向工程

安装

bash

核心库（仅解析器）


pip install scrapling

带获取器（HTTP + 浏览器自动化）- 推荐

pip install scrapling[fetchers] scrapling install

带Shell（CLI工具）- 推荐

pip install scrapling[shell]

带AI（MCP服务器）- 可选

pip install scrapling[ai]

全部安装

pip install scrapling[all]

用于隐身/动态模式的浏览器

playwright install chromium

用于Cloudflare绕过（高级）

pip install cloudscraper

代理指令

何时使用Scrapling

使用Scrapling的场景：

  • - 从网站研究主题
  • 从博客、新闻网站、文档中提取数据
  • 使用Spider爬取多个页面
  • 收集内容用于摘要
  • 从任何网站提取品牌数据
  • 从网站逆向工程API

不要用于：

  • - X/Twitter（使用x-tweet-fetcher技能）
  • 需要登录的网站（除非提供了凭证）
  • 付费墙内容（遵守robots.txt）
  • 在其服务条款中禁止抓取的网站


快速命令

1. 基本获取（最常用）

python
from scrapling.fetchers import Fetcher

page = Fetcher.get(https://example.com)

提取内容

title = page.css(h1::text).get() paragraphs = page.css(p::text).getall()

2. 隐身获取（反机器人/Cloudflare）

python
from scrapling.fetchers import StealthyFetcher

StealthyFetcher.adaptive = True
page = StealthyFetcher.fetch(https://example.com, headless=True, solve_cloudflare=True)

3. 动态获取（完整浏览器自动化）

python
from scrapling.fetchers import DynamicFetcher

page = DynamicFetcher.fetch(https://example.com, headless=True, network_idle=True)

4. 自适应解析（应对设计变更）

python
from scrapling.fetchers import Fetcher

page = Fetcher.get(https://example.com)

首次抓取 - 保存选择器

items = page.css(.product, auto_save=True)

之后 - 如果网站变更，使用adaptive=True重新定位

items = page.css(.product, adaptive=True)

5. Spider（多页面）

python
from scrapling.spiders import Spider, Response

class MySpider(Spider):
name = demo
start_urls = [https://example.com]
concurrent_requests = 3

async def parse(self, response: Response):
for item in response.css(.item):
yield {item: item.css(h2::text).get()}

# 跟随链接
next_page = response.css(.next a)
if next_page:
yield response.follow(next_page[0].attrib[href])

MySpider().start()

6. CLI使用

bash

简单获取到文件


scrapling extract get https://example.com content.html

隐身获取（绕过反机器人）

scrapling extract stealthy-fetch https://example.com content.html

交互式Shell

scrapling shell https://example.com

常见模式

提取文章内容

python
from scrapling.fetchers import Fetcher

page = Fetcher.get(https://example.com/article)

尝试多个选择器获取标题

title = ( page.css([itemprop=headline]::text).get() or page.css(article h1::text).get() or page.css(h1::text).get() )

获取段落

content = page.css(article p::text, .article-body p::text).getall()

print(f标题：{title})
print(f段落数：{len(content)})

研究多个页面

python
from scrapling.spiders import Spider, Response

class ResearchSpider(Spider):
name = research
start_urls = [https://news.ycombinator.com]
concurrent_requests = 5

async def parse(self, response: Response):
for item in response.css(.titleline a::text).getall()[:10]:
yield {title: item, source: HN}

more = response.css(.morelink::attr(href)).get()
if more:
yield response.follow(more)

ResearchSpider().start()

爬取整个网站（简易模式）

通过跟随内部链接自动爬取域名下的所有页面：

python
from scrapling.spiders import Spider, Response
from urllib.parse import urljoin, urlparse

class EasyCrawl(Spider):
自动爬取域名下的所有页面。

name = easy_crawl
start_urls = [https://example.com]
concurrent_requests = 3

def init(self):
super().init()
self.visited = set()

async def parse(self, response: Response):
# 提取内容
yield {
url: response.url,
title: response.css(title::text).get(),
h1: response.css(h1::text).get(),
}

# 跟随内部链接（限制为50页）
if len(self.visited) >= 50:
return

self.visited.add(response.url)

links = response.css(a::attr(href)).getall()[:20]
for link in links:
full_url = urljoin(response.url, link)
if full_url not in self.visited:
yield response.follow(full_url)

使用

result = EasyCrawl() result.start()

站点地图爬取

从sitemap.xml爬取页面（回退到链接发现）：

python
from scrapling.fetchers import Fetcher
from scrapling.spiders import Spider, Response
from urllib.parse import urljoin, urlparse
import re

def getsitemapurls(url: str, max_urls: int = 100) -> list:
从sitemap.xml提取URL - 同时检查robots.txt。

parsed = urlparse(url)
base_url = f{parsed.scheme}://{parsed.netloc}

sitemap_urls = [
f{base_url}/sitemap.xml,
f{base_url}/sitemap-index.xml,
f{baseurl}/sitemapindex.xml,
f{base_url}/sitemap-news.xml,
]

all_urls = []

# 首先检查robots.txt中的站点地图URL
try:
robots = Fetcher.get(f{base_url}/robots.txt)
if robots.status == 200:
sitemapinrobots = re.findall(rSitemap:\s*(\S+), robots.text, re.IGNORECASE)
for sm in sitemapinrobots:
sitemap_urls.insert(0, sm)
except:
pass

# 尝试每个站点地图位置
for sitemapurl in sitemapurls:
try:
page = Fetcher.get(sitemap_url, timeout=10)
if page.status != 200:
continue

text = page.text

# 检查是否为XML
if urls = re.findall(r([^<]+), text)
allurls.extend(urls[:maxurls])
print(f在{sitemap_url}中找到{len(urls)}个URL)
except:
continue

return list(set(allurls))[:maxurls]

def crawlfromsitemap(domainurl: str, maxpages: int = 50):
从站点地图爬取页面。

print(f正在获取{domain_url}的站点地图...)
urls = getsitemapurls(domain_url

标签

skill ai

通过对话安装

该技能支持在以下平台通过对话安装：

OpenClaw WorkBuddy QClaw Kimi Claude

方式一：安装 SkillHub 和技能

帮我安装 SkillHub 和 scrapling-1776206270 技能

方式二：设置 SkillHub 为优先技能安装源

设置 SkillHub 为我的优先技能安装源，然后帮我安装 scrapling-1776206270 技能

通过命令行安装

skillhub install scrapling-1776206270

下载

⬇ 下载 scrapling v1.0.8（免费）

文件大小: 9.64 KB | 发布时间: 2026-4-15 12:45

v1.0.8 最新 2026-4-15 12:45
v1.0.8 - Firecrawl-Style Crawl

相关推荐

s

self-improvement

Captures learnings, errors, and corrections to enable continuous improvement. Use when: (1) A command or operation fails unexpectedly, (2) User corrects Claude ('No, that's wrong...', 'Actually...'), (3) User requests a capability that doesn't exist, (4) An external API or tool fails, (5) Claude realizes its knowledge is outdated or incorrect, (6) A better approach is discovered for a recurring task. Also review learnings before major tasks.

⭐ 3209 ⬇ 392958
AI智能
s

self-improvement

Captures learnings, errors, and corrections to enable continuous improvement. Use when: (1) A command or operation fails unexpectedly, (2) User corrects Claude ('No, that's wrong...', 'Actually...'), (3) User requests a capability that doesn't exist, (4) An external API or tool fails, (5) Claude realizes its knowledge is outdated or incorrect, (6) A better approach is discovered for a recurring task. Also review learnings before major tasks.

⭐ 3209 ⬇ 392957
AI智能
s

self-improvement

Captures learnings, errors, and corrections to enable continuous improvement. Use when: (1) A command or operation fails unexpectedly, (2) User corrects Claude ('No, that's wrong...', 'Actually...'), (3) User requests a capability that doesn't exist, (4) An external API or tool fails, (5) Claude realizes its knowledge is outdated or incorrect, (6) A better approach is discovered for a recurring task. Also review learnings before major tasks.

⭐ 3195 ⬇ 389589
AI智能
s

self-improvement

Captures learnings, errors, and corrections to enable continuous improvement. Use when: (1) A command or operation fails unexpectedly, (2) User corrects Claude ('No, that's wrong...', 'Actually...'), (3) User requests a capability that doesn't exist, (4) An external API or tool fails, (5) Claude realizes its knowledge is outdated or incorrect, (6) A better approach is discovered for a recurring task. Also review learnings before major tasks.

⭐ 3177 ⬇ 386644
AI智能

多链集团旗下-闲社网

闲社网热线
免费联系电话
0527-80111111
            qqline

服务时间：周一到周日 8:00-24:00

  • 公众号
    闲社 闲社线报社区
关注闲社网
  • wxkf

    闲社在线客服

  • wx

    关注闲社网微信

  • app

    闲社网APP

Archiver·手机版·闲社网·闲社论坛·羊毛社区· 多链控股集团有限公司 · 苏ICP备2025199260号-1

Powered by Discuz! X5.0   © 2024-2025 闲社网·线报更新论坛·羊毛分享社区·http://xianshe.com

p2p_official_large
返回顶部