memory-compression-system

# Memory Compression System v3.0 **Integrated memory management and extreme context compression** for OpenClaw. Combines the best features of memory management and extreme compression into a single, streamlined skill with automatic scheduling. ## Quick Start ```bash # Install skill openclaw skill install memory-compression-system # Enable automatic compression (runs every 6 hours) scripts/enable.sh # Check status scripts/status.sh # Manual compression scripts/compress.sh --format ultra # Search memory scripts/search.sh "keyword" ``` ## Features ### 1. Integrated Memory Management - **Automatic compression**: Every 6 hours (00:00, 06:00, 12:00, 18:00 UTC) - **Smart cleanup**: Daily cleanup of old files - **Search functionality**: Unified search across all memory files - **Backup system**: Automatic backups before compression ### 2. Three Compression Formats - **Base64 Compact (B64C)**: Universal compatibility, ~40% reduction - **Custom Binary (CBIN)**: Optimized binary, ~70% reduction - **Ultra Compact (UCMP)**: Extreme compression, ~85% reduction, target ~150 tokens ### 3. Automated Scheduling - **Single cron job**: No overlapping schedules - **Configurable**: Adjust frequency and timing - **Reliable**: Built-in error recovery and logging - **Efficient**: Minimal resource usage ### 4. Search & Retrieval - **Unified search**: Across all memory formats - **Fast indexing**: Real-time search updates - **Context aware**: Understands compressed formats - **Export options**: Search results in multiple formats ### 5. ClawHub Ready - **Standard structure**: Compatible with ClawHub - **Easy installation**: One-command setup - **Health checks**: Built-in monitoring - **Documentation**: Comprehensive guides ## Installation ### Via ClawHub (Recommended) ```bash openclaw skill search memory-compression-system openclaw skill install memory-compression-system ``` ### Manual Installation ```bash cd /home/node/.openclaw/workspace/skills git clone [repository-url] memory-compression-system cd memory-compression-system scripts/install.sh ``` ## Usage ### Basic Operations ```bash # Enable automatic compression scripts/enable.sh # Disable automatic compression scripts/disable.sh # Check system status scripts/status.sh # Run health check scripts/health.sh ``` ### Compression Operations ```bash # Compress with auto format selection scripts/compress.sh # Compress to specific format scripts/compress.sh --format base64 scripts/compress.sh --format binary scripts/compress.sh --format ultra # Decompress files scripts/decompress.sh [filename] # List compressed files scripts/list.sh ``` ### Search Operations ```bash # Search across all memory scripts/search.sh "keyword" # Search with filters scripts/search.sh "keyword" --format ultra --date 2026-02-15 # Export search results scripts/search.sh "keyword" --export json # View search history scripts/search-history.sh ``` ### Management Operations ```bash # Cleanup old files scripts/cleanup.sh --days 30 # Backup system scripts/backup.sh # Restore from backup scripts/restore.sh [backup-file] # View logs scripts/logs.sh # Performance metrics scripts/metrics.sh ``` ## Configuration ### Main Configuration File Edit `config/default.conf`: ```bash # Compression settings COMPRESSION_ENABLED=true DEFAULT_FORMAT=ultra RETENTION_DAYS=30 MAX_COMPRESSED_FILES=100 # Cron schedule (UTC) CRON_SCHEDULE="0 */6 * * *" # Every 6 hours CLEANUP_SCHEDULE="0 4 * * *" # Daily at 04:00 # Search settings SEARCH_ENABLED=true SEARCH_INDEX_AUTO_UPDATE=true SEARCH_HISTORY_SIZE=1000 # Performance settings MAX_MEMORY_MB=100 MAX_PROCESSING_TIME_SEC=300 ``` ### Environment Variables ```bash export MEMORY_COMPRESSION_DEBUG=1 # Enable debug mode export MEMORY_COMPRESSION_QUIET=0 # Disable quiet mode export MEMORY_COMPRESSION_TEST=0 # Enable test mode ``` ## Compression Formats ### Format 1: Base64 Compact (B64C) ``` VERSION:3.0 FORMAT:B64C TIMESTAMP:2026-02-15T14:55:00Z SIZE:1024 CHECKSUM:crc32 DATA:<base64_encoded> ``` **Features**: - Human readable - Easy debugging - Universal compatibility - CRC32 integrity check ### Format 2: Custom Binary (CBIN) ``` [Magic:CBIN][Version:3][Flags:1][Size:4][Dictionary:var][Data:var][Checksum:2] ``` **Features**: - Optimized binary format - Shared string dictionary - Huffman encoding - CRC16 integrity check ### Format 3: Ultra Compact (UCMP) Target: ~150 tokens for complete context **Features**: - Extreme compression (~85% reduction) - 3-letter abbreviations - Bit packing - Delta encoding - String interning ## Automatic Scheduling ### Compression Schedule - **Every 6 hours**: 00:00, 06:00, 12:00, 18:00 UTC - **Operations**: 1. Backup current memory 2. Compress with optimal format 3. Update search index 4. Cleanup old files 5. Log results ### Cleanup Schedule - **Daily at 04:00 UTC** - **Operations**: 1. Remove files older than RETENTION_DAYS 2. Archive logs 3. Optimize search index 4. Update statistics ### Health Check Schedule - **Integrated with OpenClaw health checks** - **Operations**: 1. Check compression status 2. Verify file integrity 3. Monitor resource usage 4. Report issues ## Search System ### Search Index - **Real-time updates**: Index updates after each compression - **Multiple formats**: Searches across all compression formats - **Fast retrieval**: Optimized for speed - **Context aware**: Understands compressed content ### Search Features ```bash # Basic search scripts/search.sh "compression" # Advanced search scripts/search.sh "compression ratio" --format ultra --after 2026-02-01 # Export results scripts/search.sh "test" --export csv --output results.csv # Search history scripts/search.sh --history ``` ### Search Index Structure ```json { "version": "3.0", "last_updated": "2026-02-15T18:00:00Z", "files": [ { "filename": "memory_20260215_180000.ultra", "format": "ultra", "size": 256, "original_size": 1024, "ratio": 0.25, "keywords": ["compression", "memory", "test"], "timestamp": "2026-02-15T18:00:00Z" } ] } ``` ## Error Handling & Recovery ### Automatic Recovery - **Backup system**: Automatic backups before operations - **Transaction logs**: All operations logged - **Rollback capability**: Automatic rollback on failure - **Error notifications**: Alerts for critical issues ### Manual Recovery ```bash # Check error logs scripts/logs.sh --error # Restore from backup scripts/restore.sh latest # Repair search index scripts/repair-index.sh # Reset system scripts/reset.sh --safe ``` ### Common Issues & Solutions **Issue**: Compression fails **Solution**: Check disk space and run `scripts/repair.sh` **Issue**: Search not working **Solution**: Rebuild index with `scripts/rebuild-index.sh` **Issue**: Cron job not running **Solution**: Check with `scripts/status.sh --cron` **Issue**: Performance degradation **Solution**: Run `scripts/cleanup.sh --aggressive` ## Monitoring & Logging ### Log Files - `logs/compression.log`: Compression operations - `logs/search.log`: Search operations - `logs/error.log`: Error messages - `logs/performance.log`: Performance metrics - `logs/cron.log`: Cron job execution ### Status Monitoring ```bash # Basic status scripts/status.sh # Detailed health check scripts/health.sh # Performance metrics scripts/metrics.sh # System information scripts/info.sh ``` ### Alert System - **Email alerts**: For critical errors (if configured) - **Telegram notifications**: For important events - **Log monitoring**: Automatic log analysis - **Performance alerts**: For resource issues ## Testing ### Test Suite ```bash cd test ./run-tests.sh ``` ### Test Coverage - **Unit tests**: Format encoding/decoding - **Integration tests**: Full system integration - **Performance tests**: Speed and memory usage - **Error tests**: Failure scenarios - **Cron tests**: Automatic scheduling ### Manual Testing ```bash # Test compression scripts/test-compression.sh # Test search scripts/test-search.sh # Test cron job scripts/test-cron.sh # Test error handling scripts/test-errors.sh ``` ## Performance ### Compression Performance - **Base64**: < 50ms for 10KB - **Binary**: < 100ms for 10KB - **Ultra**: < 200ms for 10KB - **Target ratio**: 75-90% reduction ### Search Performance - **Indexing**: < 100ms per file - **Search**: < 50ms for 1000 files - **Memory**: < 10MB for index - **Updates**: Real-time ### Resource Usage - **CPU**: Minimal impact - **Memory**: < 50MB peak - **Disk**: Configurable retention - **Network**: Local only ## Updates & Maintenance ### Update Procedure ```bash # Update via ClawHub openclaw skill update memory-compression-system # Manual update scripts/update.sh # Check for updates scripts/check-updates.sh ``` ### Backup Recommendations - **Regular backups**: Weekly backups of `data/` directory - **Export search index**: Monthly export of search index - **Archive logs**: Monthly log archiving - **Configuration backup**: Backup `config/` after changes ### Version History - **3.0.0**: Integrated memory-compression-system (current) - **2.0.0**: Extreme context compression skill - **1.0.0**: Memory manager skill ## Contributing ### Development Setup ```bash # Clone repository git clone [repo-url] # Install dependencies npm install # Run tests npm test # Build package npm run build ``` ### Code Standards - **Bash scripts**: Follow Google Bash Style Guide - **JavaScript**: Follow Standard JS style - **Documentation**: Keep SKILL.md updated - **Testing**: Add tests for new features ### Pull Request Process 1. Fork the repository 2. Create feature branch 3. Add tests for new functionality 4. Update documentation 5. Submit pull request ## Support ### Documentation - **SKILL.md**: This file - **README.md**: User documentation - **examples/**: Usage examples - **test/**: Test examples ### Troubleshooting 1. Check `logs/` directory 2. Run `scripts/diagnose.sh` 3. Review `examples/troubleshooting.md` 4. Contact maintainer if needed ### Community - **ClawHub**: Skill repository - **Discord**: OpenClaw community - **GitHub**: Issue tracker - **Documentation**: OpenClaw docs ## License MIT License - See LICENSE file for details. --- **Note**: This skill is designed for OpenClaw context optimization. Always maintain backups of important data and test in a safe environment before production use.