NLP — Natural Language Processing Toolbox

A pure-bash NLP toolkit for text analysis. Tokenize text, analyze sentiment, extract named entities, summarize documents, compute text similarity, and classify text into categories — all from the command line with no external dependencies.

Commands

tokenize

Split text into words and sentences. Returns word count, sentence count, individual tokens, and the top 10 most frequent words.

CODEBLOCK0

sentiment

Analyze text sentiment using built-in positive/negative word lists. Returns polarity (positive/negative/neutral), a score from -1.0 to 1.0, confidence level, and matched word counts. Handles negators (e.g., "not good" flips sentiment) and intensifiers.

CODEBLOCK1

extract

Extract named entities from text: names/people (consecutive capitalized words), organizations (with suffixes like Inc, Corp, Ltd, LLC), dates (multiple formats), numbers with units, email addresses, and URLs.

CODEBLOCK2

summarize

Generate a summary by extracting the most important sentences. Scores sentences by word frequency with position bonuses (first/last sentences weighted higher). Control output length with --sentences N or --ratio 0.3.

CODEBLOCK3

similarity

Compute similarity between two texts using Jaccard index (word set overlap) and cosine similarity (word frequency vectors). Returns an overall score (average of both), shared word count, and unique word count. Scale: 0.0 = completely different, 1.0 = identical.

CODEBLOCK4

classify

Classify text into user-provided categories using keyword matching. Has built-in keyword dictionaries for common categories: finance, sports, tech, politics, science, health, positive, negative, neutral. Returns the predicted category with confidence scores and hit counts for each category.

CODEBLOCK5

Global Flags

Input Methods

All commands accept input via three methods:

1. 1. --input "text" — inline text string
2. --file path.txt — read from a file
3. Pipe via stdin — INLINECODE5

Data Storage

This tool is stateless — it does not write to disk. All processing happens in memory and output goes to stdout/stderr.

Requirements

• - Bash 4+ (uses associative arrays)
• INLINECODE6 with -P (Perl regex) for entity extraction
• INLINECODE8 for floating-point calculations
• No Python, no external NLP libraries — pure shell

When to Use

1. 1. Quick text analysis — tokenize a document to get word counts and frequency distributions without leaving the terminal
2. Sentiment checking — analyze customer reviews, social media posts, or feedback files for positive/negative polarity
3. Entity extraction — pull out names, organizations, dates, emails, and URLs from unstructured text
4. Document summarization — distill long articles or reports into key sentences at a chosen ratio
5. Text comparison — measure how similar two documents are using Jaccard and cosine metrics for deduplication or plagiarism detection

Examples

CODEBLOCK6

Output

Plain text by default with clear section headers. Use --json flag for machine-readable JSON output suitable for piping into jq or other tools. Sentiment returns polarity and score. Extract returns categorized entity lists. Similarity returns a 0.0–1.0 score.