Web + Desktop Automation

Use this skill when a task may involve:

• - Opening or controlling websites
• Reading or extracting page content
• Filling forms, clicking buttons, logging in
• Downloading or uploading files
• Controlling desktop apps with mouse/keyboard
• Combining browser steps with local app steps

Core rule

Prefer the simplest reliable path:

1. 1. If the task can be done in the browser, use browser automation.
2. If the task needs local apps or OS-level interaction, use desktop automation.
3. If both are needed, split the job into clear phases and verify after each phase.

Execution strategy

1) Classify the task

• - Browser only
• Desktop only
• Mixed browser + desktop

2) Browser automation

• - Navigation
• Search
• Page reading
• Form filling
• Clicking controls
• File upload/download
• Logged-in web workflows

Prefer stable selectors and explicit waits. Avoid brittle coordinate-based clicking when browser selectors exist.

3) Desktop automation

• - Native apps
• Window switching
• Copy/paste between apps
• File manager operations
• UI flows outside the browser

Prefer application/window-aware methods when available. Use image-based or coordinate-based control only when necessary.

4) Mixed workflows

• - Browser phase
• Desktop phase
• Browser phase again if needed

After each phase, verify the result before continuing.

Recovery rules

If a step fails:

1. 1. Re-check the current UI state
2. Re-locate the target element or window
3. Try a more stable selector or a different interaction method
4. If the task risks loss of data or irreversible action, stop and ask the user

Best practices

• - Prefer deterministic steps over guessing
• Avoid rapid blind retries
• Capture key state when tasks are long or fragile
• Keep flows small and modular
• Use scripts for repeated actions
• Use scripts/browser_runner.py for Playwright browser automation templates
• Use scripts/desktop_runner.py for PyAutoGUI desktop automation templates
• Use scripts/mixed_orchestrator.py for browser + desktop handoffs
• Put browser-specific patterns in INLINECODE3
• Put desktop-specific patterns in INLINECODE4
• Put mixed-flow orchestration examples in INLINECODE5
• Put dependency and installation notes in INLINECODE6
• Put a realistic browser-download → desktop-edit → browser-upload flow in INLINECODE7
• See requirements.txt for a minimal install set
• Put dependency and installation notes in INLINECODE9
• Put a realistic browser-download → desktop-edit → browser-upload flow in INLINECODE10
• Put dependency and installation notes in INLINECODE11