ETL
Extract-Transform-Load data toolkit (v2.0.0). Record and manage data pipeline activities across the full ETL lifecycle — ingest, transform, query, filter, aggregate, visualize, export, sample, schema definition, validation, pipeline orchestration, and data profiling. Each command logs timestamped entries to its own log file, giving you a structured record of all data operations.
Commands
| Command | Description |
|---|
| INLINECODE0 | Record a data ingestion event (source, format, row count, etc.). Without args, shows recent ingest entries. |
| INLINECODE1 |
Log a transformation step (column rename, type cast, normalization, etc.). Without args, shows recent transforms. |
|
etl query <input> | Record a query operation or SQL statement. Without args, shows recent queries. |
|
etl filter <input> | Log a filtering rule or condition applied to data. Without args, shows recent filters. |
|
etl aggregate <input> | Record an aggregation step (GROUP BY, SUM, AVG, etc.). Without args, shows recent aggregations. |
|
etl visualize <input> | Log a visualization request or chart configuration. Without args, shows recent visualizations. |
|
etl export <input> | Record an export operation (destination, format, row count). Without args, shows recent exports. |
|
etl sample <input> | Log a data sampling step (sample size, method, seed). Without args, shows recent samples. |
|
etl schema <input> | Record a schema definition or schema change. Without args, shows recent schema entries. |
|
etl validate <input> | Log a data validation rule or result. Without args, shows recent validations. |
|
etl pipeline <input> | Record a pipeline configuration or execution step. Without args, shows recent pipeline entries. |
|
etl profile <input> | Log a data profiling result (null counts, distributions, anomalies). Without args, shows recent profiles. |
|
etl stats | Show summary statistics: entry counts per category, total entries, data size, and earliest record date. |
|
etl export <fmt> | Export all logged data to a file. Supported formats:
json,
csv,
txt. (Note: this is a different code path from the
export log command — it exports the tool's own data.) |
|
etl search <term> | Search across all log files for a keyword (case-insensitive). |
|
etl recent | Show the 20 most recent entries from the activity history log. |
|
etl status | Health check: version, data directory, total entries, disk usage, last activity. |
|
etl help | Show the built-in help with all available commands. |
|
etl version | Print the current version (v2.0.0). |
Data Storage
All data is stored as plain-text log files in ~/.local/share/etl/:
- - Per-command logs — Each command (ingest, transform, query, etc.) writes to its own
.log file (e.g., ingest.log, transform.log). - History log — Every operation is also appended to
history.log with a timestamp and command name. - Export files — Generated in the same directory as
export.json, export.csv, or export.txt.
Entries are stored in timestamp|value format, making them easy to grep, parse, or pipe into downstream tools.
Requirements
- - Bash 4.0+ (uses
set -euo pipefail) - coreutils —
date, wc, du, head, tail, grep, basename, INLINECODE40 - No external dependencies, API keys, or network access required
- Works fully offline on any POSIX-compatible system
When to Use
- 1. Logging data pipeline steps — Record each stage of your ETL process (ingest → transform → validate → export) with timestamps, creating a complete audit trail of data movements.
- Schema management and validation — Use
schema to document table structures and validate to log data quality rules and their pass/fail results. - Data profiling and exploration — Use
profile to record column statistics, null rates, and distribution anomalies; use sample to log sampling parameters for reproducibility. - Pipeline orchestration tracking — Use
pipeline to record multi-step workflow configurations, execution order, and dependencies between ETL stages. - Cross-team data operations review — Run
stats for aggregate counts, search to find specific operations by keyword, and export json to share pipeline logs with team members or load into dashboards.
Examples
CODEBLOCK0
Tips
- - Run any data command without arguments to see recent entries (e.g.,
etl ingest shows the last 20 ingest entries). - Use
etl recent for a quick overview of all activity across all categories. - Combine with cron to auto-log pipeline runs: INLINECODE51
- Back up your data by copying
~/.local/share/etl/ to your preferred backup location.
Powered by BytesAgain | bytesagain.com | hello@bytesagain.com
