Desktop Automation

Yukkai Pilot provides full control of your Mac desktop through the macOS Accessibility API and screen capture. It can see your screen, understand UI elements semantically, interact with any app, and automate complex workflows — all without external dependencies.

Overview

The desktop tool follows an OBSERVE → DECIDE → ACT → VERIFY loop for every interaction. This ensures reliable, verifiable automation.

Step	Action	Purpose
1. Observe	screenshot and/or inspect	See the current UI state
2. Decide	Analyze the output	Pick the target element and action
3. Act	click, type, press, etc.	Execute exactly one action
4. Verify	screenshot again	Confirm the action had the expected effect

---

Actions

Vision

Action	Parameters	Description
screenshot	app (optional), max_dimension (optional)	Capture the full screen or a specific app window via ScreenCaptureKit.
inspect	app (optional), max_depth (default 5), format (text/json), role	Read the accessibility tree (AX Tree) — roles, labels, values, positions.

Mouse

Action	Parameters	Description
click	x, y	Click at coordinates.
right_click	x, y	Right-click at coordinates.
double_click	x, y	Double-click at coordinates.
move_mouse	x, y	Move the cursor to coordinates.
drag	x, y, to_x, to_y	Drag from one position to another.

Keyboard

Action	Parameters	Description
type	text	Type text (into the currently focused field).
press	key, modifiers (optional)	Press a key. Modifiers: cmd, shift, alt, ctrl.
shortcut	keys (e.g., "cmd+c")	Execute a keyboard shortcut.

Scroll

Action	Parameters	Description
scroll	x, y, delta_x, delta_y	Scroll at a position. Positive delta_y = up, negative = down.

Semantic Targeting (Recommended)

Action	Parameters	Description
find_and_click	label, app (optional), role (optional)	Find an element by its label and click it.
find_elements	label, app (optional), role (optional)	Find UI elements by label.
type_in_field	label, text, app (optional)	Type text into a specific field by label.

Window & App Management

Action	Parameters	Description
list_windows	—	List all open windows.
activate_app	app	Bring an app to the foreground.
launch_app	app	Launch an application.

---

Strategy

1. Start with inspect — it's faster and more precise than vision. The accessibility tree gives you semantic information (labels, roles, values) that screenshots can't provide.
2. Use find_and_click over click(x, y) — labels are stable across window resizes and repositioning. Fixed coordinates are not.
3. Use screenshot only when inspect is insufficient — e.g., for canvas-based apps, games, or non-accessible UI.
4. Verify after each action — re-inspect or re-screenshot to confirm before issuing the next action.
5. Ask for confirmation before irreversible actions (deleting files, sending emails, etc.).

---

Requirements

• Accessibility permission — required for click, type, press, inspect, find_and_click, and all interaction actions. If actions appear to do nothing, check this permission first.
• Screen Recording permission — required for screenshot. Without it, screenshots will be blank.

Both permissions are requested automatically on first use if not yet granted.

---

Examples

Click a Button by Label

desktop(action: "find_and_click", label: "Save", app: "TextEdit")
desktop(action: "screenshot", app: "TextEdit")   // VERIFY

Full Interaction Loop

desktop(action: "screenshot")                          // OBSERVE
desktop(action: "inspect", app: "Finder", max_depth: 3) // OBSERVE (semantic)
desktop(action: "find_and_click", label: "New Folder") // ACT
desktop(action: "screenshot")                           // VERIFY

Type into a Specific Field

desktop(action: "type_in_field", label: "Search", text: "quarterly report", app: "Notes")

Keyboard Shortcut

desktop(action: "shortcut", keys: "cmd+shift+s")

Launch and Switch Apps

desktop(action: "launch_app", app: "Safari")
desktop(action: "activate_app", app: "Xcode")

Inspect the Accessibility Tree

desktop(action: "inspect", app: "Safari", max_depth: 4, format: "text")

---

Common Pitfalls

• ❌ Clicking on coordinates from an old screenshot after the window moved or resized. Always take a fresh screenshot before coordinate-based clicks.
• ❌ Typing without confirming focus — use inspect or screenshot to verify the target field has focus before typing.
• ❌ Retrying a failed action in a loop without checking permissions or taking a new screenshot. If it fails once, investigate before retrying.
• ❌ Using type without first clicking the field — text may land in the wrong place or nowhere.
• ❌ Chaining multiple blind actions without a verify step — UI state can change unexpectedly (dialogs, focus changes, animations).