Web extraction for LLMs and agents. Scrape, crawl, map, search, extract, summarize, diff, monitor, and research any URL into clean Markdown, text, or JSON, i...
3 次安装
文档
web-snippet-extractor
试用Extract reusable code snippets, API patterns, and configuration examples from web pages. Use when you find a documentation page, blog post, or tutorial and need to pull out the actionable code blocks, CLI commands, or config snippets without manually scanning the entire page.
它能做什么
Extract reusable code snippets, API patterns, and configuration examples from web pages. Use when you find a documentation page, blog post, or tutorial and need to pull out the actionable code blocks, CLI commands, or config snippets without manually scanning the entire page.
技能文档
Web Snippet Extractor
Extract clean, copy-paste-ready code snippets from any web page.
When to Use
- You found a documentation page and want just the code, not the prose
- You need to collect CLI commands from a tutorial
- You want to extract API examples or configuration blocks from a blog post
- You're building a reference library and need structured snippets with context
Prerequisites
web_fetchtool available (built into OpenClaw)- Target URL must be publicly accessible
Steps
1. Fetch the page content
web_fetch(url="", extractMode="markdown", maxChars=20000)
Use markdown mode to preserve code block delimitings (lang ... ).
2. Identify and extract snippets
Look for fenced code blocks in the markdown output. Each snippet should include:
- Language (the tag after the opening ```)
- Title/context (the heading or paragraph immediately preceding the block)
- Body (the code content itself)
3. Classify snippets
Tag each extracted snippet:
| Tag | Meaning |
|---|---|
config | Configuration files (yaml, toml, json, env) |
command | CLI/shell commands |
code | Source code (any language) |
script | Full automation scripts (bash, python, etc.) |
snippet | Short inline patterns or one-liners |
4. Output format
Return snippets as structured blocks:
### Snippet 1:
- **Language:**
- **Type:**
5. Save to workspace (optional)
If the user wants to persist snippets, write them to a file:
workspace/snippets/.md
Example
Input: A URL to a Docker documentation page about docker compose
Output:
### Snippet 1: Start services in detached mode
- **Language:** bash
- **Type:** command
```bash
docker compose up -d
Snippet 2: docker-compose.yml for a web app
- Language: yaml
- Type: config
services:
web:
image: nginx:alpine
ports:
- "80:80"
## Notes
- Always include the source URL in the output header
- If a page has >10 snippets, summarize the top 5 most relevant by default and offer to list more
- Strip out boilerplate (license headers, import noise) when the user asks for "clean" snippets
- Respect the page's content license; mention attribution when requested
相关技能
Extract structured data from HTML pages or URLs using CSS selectors, XPath, regex, or LLM natural...
1 次安装
Fetch any public web page and get back clean, LLM-ready markdown (polite, robots-respecting)
Extract and clean readable article content, metadata, and markdown from URLs or HTML for research, note taking, and web scraping.
66 次安装1 星标
Summarize any web page by URL or by pasting text. Returns a concise structured summary with key points, entities, and action items. Use when the user wants a quick digest of an article, documentation page, or any web content without reading the full source.
Deep-crawl any website from start URLs, return per-page LLM-ready text/markdown/HTML plus metadata (title, description, author, language, canonical URL, OG) and in-scope outbound links. Use when user mentions deep crawl website, recursive crawl, crawl a whole site, scrape entire website, scrape docs