PyAutoGUI Automation Skill

Cross-platform mouse/keyboard automation for Windows, Linux, and macOS.

Features

• - Mouse control: move, click, drag, scroll
• Keyboard control: key press, hotkeys, type text
• Screen operations: screenshot, mouse position, screen size
• Image utilities: image metadata (size/format/file size), crop images
• Screen overlay: draw temporary markers to validate coordinates
• Draw on images: draw persistent markers into an image and save
• Image locating: template matching and OCR-based text locating
• Cleanup: remove generated screenshots/marked files to free disk space

Activation

Activate when the user asks to do things like:

• - "Click a position on the screen"
• "Move the mouse to (x, y)"
• "Type text / press keys"
• "Take a screenshot"
• "Run repetitive UI automation"
• "Get the current mouse position"
• "Get image size / image info"
• "Crop an image"
• "Draw a marker on the screen"
• "Draw a marker on an image"
• "Locate an element by template"
• "Locate text on the screen (OCR)"
• "Clean up screenshots / temporary files"

Usage

Install dependencies

CODEBLOCK0

Screen info

CODEBLOCK1

Mouse actions

CODEBLOCK2

Keyboard actions

CODEBLOCK3

Screenshot

CODEBLOCK4

Screenshot notes:

• - Supported formats: PNG (recommended), JPG, BMP, etc.
• Scope: primary monitor (in multi-monitor setups)

Region Screenshot

CODEBLOCK5

Parameters:

• - x1, y1: Top-left corner coordinates
• INLINECODE1: Bottom-right corner coordinates
• Order doesn't matter (automatically calculated)

Copy & Paste

CODEBLOCK6

Use cases:

• - copy_paste is faster than type_text for long text
• Use copy_paste when you want to skip typing animation
• Use type_text when you need to simulate realistic typing

Common key names

• - Letters: a b c ...
• Numbers: 0 1 2 ...
• Function keys: f1 f2 ... INLINECODE14
• Modifiers: ctrl alt shift INLINECODE18
• Others: enter esc tab space backspace delete up down left INLINECODE28

Safety

⚠️ Important:

1. 1. Make sure the target window is focused before executing actions
2. Be careful with system hotkeys to avoid unintended actions
3. Add delays when needed to give yourself time to interrupt
4. Moving the mouse to the top-left corner (0, 0) triggers PyAutoGUI failsafe

Cross-platform notes

• - Windows: Full support; admin permission may be needed in some environments
• Linux: Requires X11; Wayland may not work
• macOS: Grant Accessibility permission to Terminal/Python in System Settings

Example scenarios

Open Calculator (Windows)

Auto-fill a form

Batch clicking

Included scripts

• - scripts/keyboard_mouse.py - Mouse/keyboard control
• INLINECODE30 - Image utilities
• INLINECODE31 - Screen overlay markers
• INLINECODE32 - Draw markers on images
• INLINECODE33 - Image locating (template + OCR)
• INLINECODE34 - Cleanup tool

Image utilities

Image info

CODEBLOCK10

Crop image

CODEBLOCK11

Output example

CODEBLOCK12

Image fields

Coordinate system

Screen coordinates:

• - Origin (0, 0) is the top-left corner
• X increases to the right
• Y increases downward

Crop coordinates:

• - x1, y1: top-left corner of crop
• INLINECODE42: bottom-right corner of crop
• Cropped size = (x2 - x1) × (y2 - y1)

Example:
CODEBLOCK13

Typical workflows

Analyze positions in a screenshot

Batch image sizing

Capture a region of the screen

python3 scripts/keyboard_mouse.py screenshot full.png
python3 scripts/image_utils.py crop full.png 500 300 1000 800 -o region.png

Screen overlay markers

Draw temporary markers on the screen for coordinate verification. Useful for:

• - Calibrating coordinates
• Confirming the real position of a button/element
• Debugging automation scripts

Draw a marker

CODEBLOCK17

Draw a rectangular area

CODEBLOCK18

Marker types

Colors

INLINECODE48, green, blue, yellow, cyan, magenta, white, INLINECODE55

Coordinate calibration example

CODEBLOCK19

Draw markers on images

Draw persistent markers into image files. Useful for:

• - Annotating recognized positions on a screenshot
• Producing reference images
• Batch marking candidates for comparison
• Keeping calibration records

Draw a marker

CODEBLOCK20

Draw a rectangular area

CODEBLOCK21

Batch marking workflow

CODEBLOCK22

Screen overlay vs drawing on image

Recommended coordinate calibration (cost-saving)

CODEBLOCK23

Image locating

Built on OpenCV template matching and RapidOCR. Supports locating UI elements by image and by text.

Install dependencies

CODEBLOCK24

Note: RapidOCR models are ~15MB and are downloaded automatically on first use.

Template matching (find by image)

CODEBLOCK25

Output example:
CODEBLOCK26

OCR text locating (find by text)

CODEBLOCK27

Output example:
CODEBLOCK28

Recommended automation workflows

Template matching (most accurate):
CODEBLOCK29

OCR text locating (when no template is available):
CODEBLOCK30

Important principle:

1. 1. OCR returns accurate screen coordinates; do not modify the returned coordinates
2. If there are multiple candidates, mark them on an image to visually choose the correct one
3. Once you choose the right candidate, click using the original coordinates

Template matching vs OCR

Why this is better than guessing coordinates

1. 1. High precision and repeatability (pixel-level)
2. Local compute with no API cost
3. Fast response
4. Easy to debug via marked outputs

Cleanup

Analyze disk usage

CODEBLOCK31

Clean files

CODEBLOCK32

Auto cleanup

CODEBLOCK33

End-to-end example

CODEBLOCK34

Command quick reference

Mouse/keyboard (keyboard_mouse.py)

Image utilities (image_utils.py)

Screen overlay (draw_overlay.py)

Draw on image (draw_on_image.py)

Image finder (image_finder.py)

Cleanup (cleanup.py)