OpenClaw VLN Planner

Use this skill when the user wants a robot to follow a natural-language navigation instruction from visual observations.

This skill is a high-level navigation planner. It does not produce motor, joint, torque, or trajectory control. It only produces one structured mid-level navigation action at a time.

When this skill triggers

Trigger this skill when the task includes one or more of the following:

• - Vision-language navigation (VLN)
• Robot next-step planning from camera images
• Closed-loop navigation with replanning after each observation
• Converting a current frame plus historical frames into a single next navigation action
• Sending current + history images to an OpenAI-compatible multimodal gateway for action prediction

Required inputs

The planner expects:

• - user_instruction: natural-language navigation instruction
• INLINECODE1: exactly one current image
• INLINECODE2: zero or more previous images in temporal order

Optional inputs:

• - robot_state: heading, speed, pose estimate, execution feedback, etc.
• INLINECODE4: blocked, collisionrisk, lost, targetreached, low_visibility, etc.
• INLINECODE5: path to the runtime config file

Output contract

Output must be pure JSON only. Do not prepend or append prose.

Allowed action types only:

• - INLINECODE6
• INLINECODE7
• INLINECODE8
• INLINECODE9

Expected JSON shape:

CODEBLOCK0

Completion example:

CODEBLOCK1

Core rules

1. 1. Plan only the next action.
2. Never output a full route.
3. Replan after each execution step.
4. If uncertain, unsafe, blocked, unable to parse, or visually ambiguous, output STOP.
5. Enforce action bounds:

1. 6. If safety_flags.target_reached == true, output STOP with task_status = completed.
2. If blocked, collision_risk, lost, or severe uncertainty is present, prefer STOP.

Runtime configuration

Before running, load a YAML config file such as config/vln-config.yaml.

The config should define:

• - subscribed or logical input topics / channels for current frame and history frame collection
• optional robot state and safety flag sources
• OpenAI-compatible multimodal gateway settings: base_url, api_key, INLINECODE25
• planner behavior such as confidence threshold and safety fallback
• executor bridge mode (default: Python function bridge)

Read references/navigation-schema.md for the expected config structure.

Internal module design

1) context builder

• - user instruction
• historical observations
• current observation
• optional robot state
• optional safety flags

The prompt must explicitly separate:

• - historical observations
• current observation
• user instruction

2) action planner

• - one current image
• historical images
• planner prompt
• optional structured context

The model should be asked to return pure JSON for exactly one next action.

3) action parser

If parsing fails:

• - try safe extraction of the first JSON object
• if still invalid, fall back to INLINECODE27

4) action validator

• - action type is one of the four allowed values
• distance and angle ranges are legal
• unit matches action type
• confidence is numeric if present
• task_status is one of in_progress, completed, INLINECODE30

Any invalid output falls back to STOP.

5) executor bridge

Reserved Python bridge interface:

• - INLINECODE32
• INLINECODE33
• INLINECODE34
• INLINECODE35
• INLINECODE36
• INLINECODE37

Do not hardcode a robot SDK into the planner logic.

6) replanning loop

1. 1. gather current frame + history frames
2. gather optional robot state / safety flags
3. call multimodal planner
4. parse and validate JSON action
5. execute through bridge
6. observe again
7. repeat until task_status = completed or forced stop

7) safety fallback

• - parse failure
• invalid action
• confidence below threshold
• blocked / collision risk / lost / target reached
• missing visual evidence for safe motion

Prompt template

Use this prompt pattern:

CODEBLOCK2

Example user requests

• - "Go down the hallway and stop at the blue door."
• "Move to the kitchen entrance."
• "Find the end of the corridor and stop."
• "Turn right at the next intersection and continue."

Failure handling

If anything is wrong with the output, return:

CODEBLOCK3

Bundled resources

• - references/navigation-schema.md: schema, bounds, safety fallback, examples, config contract
• INLINECODE40: example OpenAI-compatible multimodal planner + Python executor bridge
• INLINECODE41: Python dependencies
• INLINECODE42: runtime config template