model-tester模型测试器

Use scripts/model_tester.py to run repeatable test prompts and compare requested vs actual model usage from OpenClaw logs.

Run

From the skill directory (or pass absolute paths):

CODEBLOCK0

Inputs

• - --agent <name>: Target agent (chat, menial, coder, etc.)
• INLINECODE2: Requested model alias/name to test
• INLINECODE3: Case from references/test-cases.json or INLINECODE5
• INLINECODE6: Per-case timeout (default 120)
• INLINECODE8: Optional JSON output file

Require at least one of --agent or --model.

What the runner does

1. 1. Load test cases from references/test-cases.json.
2. Start openclaw logs --follow --json in parallel.
3. Run openclaw agent --json with a bounded test prompt (asks agent to use a subagent for the task).
4. Parse response + tailed logs.
5. Emit machine-readable JSON and a short human summary.

Output format

Top-level JSON:

• - INLINECODE14
• INLINECODE15
• INLINECODE16
• INLINECODE17
• INLINECODE18

Each result entry returns:

• - INLINECODE19
• INLINECODE20
• INLINECODE21
• INLINECODE22 (parsed from logs when available)
• INLINECODE23 (ok/error)
• INLINECODE26
• INLINECODE27
• INLINECODE28 (when discoverable)
• INLINECODE29

Privacy & Safety

The tester spawns isolated subagent tasks with predefined test prompts only — no user data is passed to models. It tails OpenClaw logs to extract:

• - which model was actually selected (routing validation)
• token usage statistics
• runtime metrics

Log extraction uses regex patterns to find model/token fields. No personally identifiable information or arbitrary log content is captured — only structured fields related to the test execution.

Notes

• - Model extraction and token extraction are best-effort because log fields may vary by OpenClaw/provider version.
• If openclaw config is invalid or gateway is unavailable, the script returns status=error with stderr details.
• Edit references/test-cases.json to add custom prompts for your benchmark set.
• All test cases are generic; no workspace or user data is baked in.