Web Autopilot

Record once in any web app, let AI handle it from now on.

Overview

Record → Analyze → Confirm Fields → Generate → Test → Register as Tool

CODEBLOCK0

Task Types

📊 Query / Export

Data extraction and report generation. Scripts run and output results automatically — no manual intervention needed. Examples: pull sales reports, extract project data, export revenue details

📝 Submit

Submit forms such as expense reports, travel requests, payment requests, etc. Each run requires dynamic parameters. Examples: submit travel request, submit expense report, submit payment request

The key challenge for submit tasks: correctly distinguishing which fields are fixed vs. which change every time, and confirming with the user before generating the script.

Skill Directory

CODEBLOCK1

Skill scripts: /opt/homebrew/lib/node_modules/openclaw/skills/web-autopilot/scripts/

---

Commands

1. record — Record a workflow

Ask user: task name, login URL or app URL.

CODEBLOCK2

Run in PTY mode (pty: true, background: true). User operates browser, types "done" when finished.

Note: --sso-url is a legacy parameter name; it works for any login URL (SSO, OAuth, plain login page, etc.).

2. analyze — Analyze the recording (AI does this)

Read recording.json, separate login traffic from business traffic, identify core APIs.

Key steps:

1. 1. Read ~/.openclaw/rpa/recordings/<task>/summary.txt for overview
2. Parse recording.json to extract all API calls to app domain
3. For each POST/PUT/PATCH with meaningful body:

- Classify fields: FIXED / DYNAMIC / SESSION / RELATIONAL
- Detect protocol: rest-json / graphql / form-urlencoded / multipart

1. 4. Map the complete API sequence (prerequisites → main operation → follow-ups)
2. Analyze ALL response fields and create field-mapping.json with human-readable labels
3. Create task-meta.json
4. [Submit tasks] After analysis, present the field classification confirmation table to the user (see below)

Field Classification

Type	Meaning	Handling
FIXED	Same value every submission (approval flow ID, company entity, currency, expense type enums…)	Hardcoded in script
DYNAMIC

Different each submission (amount, date, reason, attachment path…) | Becomes CLI --parameter | | SESSION | Auth tokens/cookies, auto-managed | Injected by session.ts | | RELATIONAL | Requires a lookup from another API to get the ID (e.g., project ID, person ID…) | Auto-queried in script, or exposed as DYNAMIC parameter |

Field Analysis Rules (MANDATORY)

Every field must have a human-readable label. Including system-generated field names.

Inference priority:

1. 1. Data value type: timestamp (10^12-13) / monetary amount (contextual) / enum (fixed values) / URL / JSON object
2. Field name pattern: time/date/at → datetime | amount/price/cost → monetary | id/key → ID | status/state → status
3. Business context: infer from related fields, API endpoint names
4. If uncertain → annotate as INLINECODE6

Field Confirmation Step (MANDATORY for Submit tasks)

After analysis, you must present the following confirmation table to the user and wait for confirmation before generating the script:

CODEBLOCK3

Only proceed to the Generate step after user confirmation.

CSV Export Rules (MANDATORY)

• - Keep ALL fields, including hidden fields, dynamic fields, system fields — never crop
• Field order: preserve original order from data, never sort (sorting causes column misalignment)
• JSON/object fields → convert to JSON string for storage
• Use csv.writer + proper quoting to handle JSON fields containing commas

3. generate — Generate the task script

Pre-generation checklist (Query/Export tasks):

• - ✅ All fields are in field-mapping.json
• ✅ All fields have human-readable labels
• ✅ CSV export uses field-mapping.json for column headers
• ✅ Field order preserves original order

Pre-generation checklist (Submit tasks):

• - ✅ User has confirmed field classification (FIXED / DYNAMIC / RELATIONAL)
• ✅ All DYNAMIC fields converted to CLI parameters (with type, example value, required/optional)
• ✅ RELATIONAL fields have auto-query logic or corresponding parameters
• ✅ Script has --dry-run mode (prints request body without submitting, for testing)
• ✅ Script outputs submission result (success/failure + document number/link)

Submit task invocation example (written to task-meta.json usage field after generation):
CODEBLOCK4

4. test — Iterative test loop (max 5 rounds)

Run script → check output → if error: diagnose → fix → repeat.

Error	Cause	Fix
401/403	Session expired / wrong auth	Re-check auth headers, re-login
400

Wrong field name/type | Compare with recording |
| 404 | Wrong URL | Check URL exactly |
| JSON parse error | Response is HTML | Log resp.raw |

5. run — Execute a registered task

CODEBLOCK5

6. list — List all tasks

CODEBLOCK6

---

Session & Credential Management

Session (Cookie/Token Storage)

Sessions are cookie-based and work with any login method:

• - SSO (OIDC, SAML, CAS, etc.)
• OAuth / OAuth2
• Username + password forms
• Any browser-based authentication

Session files: INLINECODE14

Credentials (Encrypted Storage)

Login credentials are stored encrypted (AES-256-GCM) in a separate file — never stored in plaintext.

File: ~/.openclaw/rpa/credentials.enc

• - Encryption key = machine identity (hostname+username) + optional RPA_CREDENTIAL_KEY env var
• File permissions: 0600 (owner only)
• Supports automatic extraction and encrypted storage from recording.json

CODEBLOCK7

Auto-Login Flow

When a session expires, the auto-login flow kicks in:

CODEBLOCK8

Login Flow Types

When generating scripts, you must identify the login type from the recording and write it to the loginFlow field in task-meta.json:

type	Scenario	Auto-login method	Example
api	SSO/app provides a REST login endpoint, single POST completes auth	Call API directly → follow redirects	Enterprise SSO (POST /api/sso/login)
form

Single-page login form (username + password on same page) | Fill form fields → click submit | Common admin dashboards |
| multi-step | Multi-step login (email → next page → password → next page → possible 2FA) | Execute step sequence | Google, Microsoft, Okta |
| manual-only | Has CAPTCHA/2FA/risk control, cannot be fully automated | Open headed browser directly | Banking systems, strong CAPTCHA sites |

loginFlow Schema (task-meta.json)

CODEBLOCK9

Login Identification Guide for Analyze Step (MANDATORY)

During the analyze step, you must complete the following login analysis:

1. 1. Extract credentials → credentials.ts extract <recording.json> (auto-detects username/password in POST body)
2. Identify login type → Inspect the login flow in the recording:

- Has a clear POST login/auth API → type = api - Has form fill actions (password type input) on the same page → type = form - Has multiple form fill actions with page navigations in between → type = multi-step - Has CAPTCHA image requests or reCAPTCHA scripts → type = manual-only

1. 3. Document the SSO → app redirect path:

- Does it use an appId forward? - Does it use a redirect_uri callback? - Where is the token — in URL query / response body / cookie?

1. 4. Write loginFlow → Write all fields to INLINECODE30
2. Sanitize → Replace passwords in recording.json with INLINECODE31

⚠️ If credentials.ts extract cannot extract credentials (e.g., Google multi-step login), prompt the user to save credentials manually:
CODEBLOCK10

Login Code Templates for Script Generation

Choose the auto-login implementation based on loginFlow.type:

type=api (REST API login):
CODEBLOCK11

type=form:
CODEBLOCK12

type=multi-step:
CODEBLOCK13

type=manual-only:
CODEBLOCK14

task-meta.json loginFlow example (SSO → enterprise app):
CODEBLOCK15

⚠️ Security Rules (MANDATORY)

1. 1. Passwords in recording.json must be sanitized immediately after analysis (replace with [REDACTED])
2. credentials.enc is an encrypted binary file — do not attempt to read or edit directly
3. credentials.enc and sessions/ directory must never be committed to version control or shared
4. Skill packages (.skill) must not contain any credentials, sessions, or recording data
5. Generated task scripts (run.ts) must never hardcode any passwords

---

Known Issues & Lessons Learned

🔐 Login flow must match app — don't assume one-size-fits-all

• - Current implemented scripts use type=api mode (enterprise SSO → business app)
• Each new app recording must re-identify the login type — do not reuse login logic from old scripts
• Google/Microsoft multi-step logins require type=multi-step + steps sequence
• Sites with CAPTCHA/2FA can only use INLINECODE36
• Inlining login logic into run.ts (rather than importing external login.ts) is more stable due to Node ESM/CJS compatibility issues

⚠️ Node v25 ESM compatibility

• - Node v25 defaults to ESM, require() is unavailable
• Solution: place tsconfig.json in the task directory to force INLINECODE39
• Dependencies like Playwright need full-path require: INLINECODE40
• Cross-directory .ts imports under ts-node are unstable — recommend inlining critical logic into run.ts

⚠️ Multi-tab traffic capture (fixed)

Some login flows or apps open new tabs. Recorder uses context.on('request/response') to capture ALL tabs.

📋 CSV must include ALL fields with human-readable labels

• - Never crop fields — include everything from the API response
• System-generated field names (e.g. field_*, attr_*, custom_*) must be analyzed from sample data
• Create field-mapping.json for every task
• Field order: preserve original order from data, never sort
• Use proper CSV quoting to handle JSON fields with commas

📝 Submit tasks: always confirm field classification before generating

• - Never skip the field confirmation step — wrong FIXED/DYNAMIC split breaks every future submission
• Fields that look fixed (e.g. a hardcoded project ID) might actually need to be dynamic in real use
• Always include --dry-run in generated scripts so users can verify the request body before committing
• RELATIONAL fields (e.g. approver ID looked up by name) should be auto-resolved in script, exposed as human-readable params

---

File Locations

Item	Path
Recorder	INLINECODE46
Task runner

scripts/run-task.ts | | Session utility | scripts/utils/session.ts | | Login helper | scripts/utils/login.ts | | Recordings | ~/.openclaw/rpa/recordings/<task>/ | | Generated tasks | ~/.openclaw/rpa/tasks/<task>/ | | Sessions | ~/.openclaw/rpa/sessions/<domain>.session.json |