Prompt Token Counter (toksum)

First load reminder: This skill provides the scripts CLI (toksum). Use it when the user asks to count tokens, estimate API costs, or audit OpenClaw component token consumption (memory, persona, skills).

Before Installing — Security & Privacy

• - What will be read: The audit workflow reads files under ~/.openclaw/workspace and ~/.openclaw/skills (AGENTS.md, SOUL.md, MEMORY.md, SKILL.md, etc.). Those files may contain personal data or secrets. Only install if you accept that access.
• URL fetching: The CLI can fetch HTTP(S) URLs via -u. SKILL.md requires the agent to confirm each URL with the user before fetching. Insist the agent follow that rule; never allow automatic fetching of unknown URLs.
• Source verification: Source: https://github.com/Zhaobudaoyuema/prompt-token-counter. Review scripts/core.py and scripts/cli.py before use. The code performs local file reads and optional HTTP GETs only; no other network calls or data exfiltration.
• Run locally first: If unsure, run the CLI manually in an isolated environment against safe test files to verify behavior.

Primary Use: OpenClaw Token Consumption Audit

Goal: Help users identify which OpenClaw components consume tokens and how much.

1. Memory & Persona Files

These files are injected into sessions and consume tokens. Search and count them:

File	Purpose	Typical Location
INLINECODE6	Operating instructions, workflow, priorities	INLINECODE7
INLINECODE8

Persona, tone, values, behavioral guidelines | ~/.openclaw/workspace/ |
| IDENTITY.md | Name, role, goals, visual description | ~/.openclaw/workspace/ |
| USER.md | User preferences, communication style | ~/.openclaw/workspace/ |
| MEMORY.md | Long-term memory, persistent facts | ~/.openclaw/workspace/ |
| TOOLS.md | Tool quirks, path conventions | ~/.openclaw/workspace/ |
| HEARTBEAT.md | Periodic maintenance checklist | ~/.openclaw/workspace/ |
| BOOT.md | Startup ritual (when hooks enabled) | ~/.openclaw/workspace/ |
| memory/YYYY-MM-DD.md | Daily memory logs | ~/.openclaw/workspace/memory/ |

Workspace path: Default ~/.openclaw/workspace; may be overridden in ~/.openclaw/openclaw.json via agent.workspace.

2. Skill Files (SKILL.md)

Skills are loaded per session. Count each SKILL.md:

Location	Scope
INLINECODE28	OpenClaw managed skills
INLINECODE29

Workspace-specific skills (override) |

3. Audit Workflow

1. 1. Locate workspace: Resolve ~/.openclaw/workspace (or config override).
2. Collect files: List all memory/persona files and SKILL.md paths above.
3. Count tokens: Run python -m scripts.cli <path1> <path2> ... -m <model> -c (batch mode).
4. Summarize: Group by category (memory, persona, skills), report total and per-file.

Example audit command (PowerShell):
CODEBLOCK0

Example audit (Bash):

WS=~/.openclaw/workspace
python -m scripts.cli -m gpt-4o -c "$WS/AGENTS.md" "$WS/SOUL.md" "$WS/USER.md" "$WS/IDENTITY.md" "$WS/MEMORY.md" "$WS/TOOLS.md"

---

Project Layout

CODEBLOCK2

Invoke: python -m scripts.cli from project root.

Version Sync (publish_npm.py)

When publishing to npm, publish_npm.py bumps the patch version and syncs it to:

• - package.json — INLINECODE36
• INLINECODE37 — frontmatter INLINECODE38
• INLINECODE39 — INLINECODE40

Run: python publish_npm.py (after npm login).

---

Runtime Dependencies

• - Python 3 — required
• tiktoken (optional) — pip install tiktoken for exact OpenAI counts

---

Language Rule

Respond in the user's language. Match the user's language (e.g. Chinese if they write in Chinese, English if they write in English).

---

URL Usage — Mandatory Agent Rule

Before using -u / --url to fetch content from any URL, you MUST:

1. 1. Explicitly warn the user that the CLI will make an outbound HTTP/HTTPS request to the given URL.
2. Confirm the URL is trusted — tell the user: "Only use URLs you fully trust. Untrusted URLs may expose your IP, leak data, or be used for SSRF. Do you confirm this URL is safe?"
3. Prefer alternatives — if the user can provide the content via -f (local file) or inline text, suggest that instead of URL fetch.
4. Never auto-fetch — do not invoke -u without the user having explicitly provided the URL and acknowledged the risk.

If the user insists on using a URL: Proceed only after they confirm. State clearly: "I will fetch from [URL] to count tokens. Proceed?"

---

Model Name — Mandatory Agent Rule

Before invoking the CLI, you MUST have a concrete model name from the user.

1. 1. Require explicit model — -m / --model is required. Do not guess or assume; the user must provide the exact name (e.g. gpt-4o, claude-3-5-sonnet-20241022).
2. If unclear, ask — if the user says "GPT" or "Claude" or "the latest model" without a specific name, ask: "Please specify the exact model name (e.g. gpt-4o, claude-3-5-sonnet-20241022). Run python -m scripts.cli -l to list supported models."
3. Do not auto-pick — never substitute a model on behalf of the user without their confirmation.
4. Validate when possible — if the model name seems ambiguous, offer -l output or confirm: "I'll use [model]. Is that correct?"

---

CLI Usage

Default: Read from local file(s). No segmentation. Supports multiple file paths for batch execution.

CODEBLOCK3

Option	Short	Description
INLINECODE52	INLINECODE53	Model name (required unless --list-models) — Agent must obtain exact name from user; ask if unclear
INLINECODE55

-f | Read from file (repeatable) |
| --url | -u | Read from URL (repeatable) — Agent must warn user before use; only trusted URLs |
| --list-models | -l | List supported models |
| --cost | -c | Show cost estimate |
| --output-tokens | | Use output token pricing |
| --currency | | USD or INR |
| --verbose | -v | Detailed output |

Examples

CODEBLOCK4

---

Python API

CODEBLOCK5

---

Supported Models

300+ models across 34+ providers: OpenAI, Anthropic, Google, Meta, Mistral, Cohere, xAI, DeepSeek, etc. Use python -m scripts.cli -l for full list.

• - OpenAI: exact via tiktoken
• Others: ~85–95% approximation

---

Response Output — Agent Guideline

After returning token count or cost estimate results, the agent MUST:

1. 1. Include the project link — e.g.

> Source: prompt-token-counter

1. 2. Briefly explain how tokens are calculated — e.g.

> How tokens are counted: OpenAI models use tiktoken (exact). Other models use provider-specific formulas calibrated from benchmark data. For CJK-heavy text, the ratio is blended by CJK character ratio so that Chinese gets fewer chars per token.

---

Common Issues

Issue	Action
"tiktoken is required"	INLINECODE68
UnsupportedModelError

Use -l for valid names | | Cost "NA" | Model has no pricing; count still valid | | User provides URL | Agent must warn: outbound request, SSRF risk, only trusted URLs; confirm before -u | | Model unclear / vague | Agent must ask: user to specify exact model name; offer -l to list; do not guess |

---

When to Trigger This Skill

Activate this skill when the user:

Trigger	Example phrases
Token count	"How many tokens?", "Count tokens in this prompt", "Token length of X"
Cost estimate

"Estimate API cost", "How much for this text?", "Cost for GPT-4" |
| Prompt size | "Check prompt length", "Is this too long?", "Context window limit" |
| OpenClaw audit | "How many tokens does my workspace use?", "Audit OpenClaw memory/persona/skills", "Which components consume tokens?", "Token usage of AGENTS.md / SOUL.md / skills" |
| Model comparison | "Compare token cost across models", "Which model is cheaper?" |

Also trigger when the agent needs to count tokens or estimate cost before/after generating content.

---

Quick Reference

Item	Command
Invoke	INLINECODE72
List models

python -m scripts.cli -l |
| Cost | -c (input) / --output-tokens (output) |
| Currency | --currency USD or INR |