BigData

A comprehensive data processing toolkit for ingesting, transforming, querying, filtering, aggregating, and managing data workflows — all from the command line with local timestamped log storage.

Commands

Each data command (ingest, transform, query, etc.) works the same way:

• - With arguments: saves the entry with a timestamp to its dedicated .log file and records it in the activity history
• Without arguments: displays the 20 most recent entries from that command's log

Data Storage

All data is stored locally in plain-text log files:

CODEBLOCK0

Each entry is stored as YYYY-MM-DD HH:MM|<value> for easy parsing and export.

Requirements

• - Bash 4.0+ (uses set -euo pipefail)
• Standard UNIX utilities: date, wc, du, grep, head, tail, INLINECODE27
• No external dependencies or API keys required
• Works offline — all data stays on your machine

When to Use

1. 1. Data pipeline tracking — Record each step of a multi-stage data workflow (ingest → transform → validate → export) with full timestamps for audit trails
2. Quick data logging — Capture observations, measurements, or notes about datasets directly from the terminal without opening a separate app
3. Schema management — Keep track of schema definitions, changes, and validation rules as your data evolves over time
4. Data quality monitoring — Log validation checks and profiling results to build a history of data quality metrics
5. Workflow documentation — Use search and recent commands to review what data operations were performed, when, and in what order

Examples

Log a complete data workflow

CODEBLOCK1

Search and review activity

CODEBLOCK2

Pipeline and profiling

CODEBLOCK3

Filter and query tracking

CODEBLOCK4

Output

All commands print confirmation to stdout. Data is persisted in ~/.local/share/bigdata/. Use bigdata stats for a summary or bigdata search <term> to find specific entries across all logs.