Computer Use Kylinos

Computer Use KylinOS

GUI automation across KylinOS V11 (Wayland) and V10SP1 (X11). Resolve script paths relative to this skill directory.

Step 0 — detect the OS

Always run first; pick the backend by result.

bash scripts/detect-os.sh   # -> v11 | v10sp1 | unknown

Fast path

V11 (Wayland) — wlcctrl

bash scripts/wlcctrl-info.sh                      # inspect once per task
wlcctrl --search ''
wlcctrl --getwindowgeometry 
wlcctrl --windowcapture  --path /tmp/win.png   # pick coords from this
wlcctrl --windowactivate 
wlcctrl --keystring 'text'                         # keyboard is fastest
wlcctrl --key ctrl+s
bash scripts/wlcctrl-window-click.sh    [button]   # scale-aware click
bash scripts/wlcctrl-window-dblclick.sh    [button] [delay_ms]  # double-click (geometry queried ONCE, no re-verify)
bash scripts/wlcctrl-drag.sh      [button] # drag

For Kaiming apps: kaiming list | grep -i ''; kaiming run .

V10SP1 (X11) — xdotool

bash scripts/xdotool-info.sh                      # inspect once per task
xdotool search --name ''
xdotool getwindowgeometry 
import -window  /tmp/win.png                  # screenshot to pick coords (needs imagemagick)
xdotool windowactivate 
xdotool type 'text'                                # keyboard is fastest
xdotool key ctrl+s
bash scripts/xdotool-window-click.sh    [button]   # click
bash scripts/xdotool-window-dblclick.sh    [button] [delay_ms]  # double-click (native --repeat 2, single process)
bash scripts/xdotool-drag.sh      [button] # drag

Find the app window, screenshot it, read target (rx,ry) from the image, then click/drag with the relative coordinates.

Universal rules

• Do not move a window to (0,0) by default. Use the move-window script only when a known position is required, the window is partly off-screen, or the user asks.
• Prefer keyboard shortcuts/input over pointer clicks when available.
• Capture the window in its current position; identify targets from the screenshot.
• For wheel/spinner controls prefer small drag gestures over scroll (scroll may jump values).
• Avoid closing/killing windows unless explicitly requested.
• Cache window geometry / display info per task — repeated probes are expensive.

V11 (wlcctrl) specifics

• ⚠️ man page is WRONG about coordinates. It documents --mousemove/--windowmove as relative displacements; both are actually ABSOLUTE. Pass the absolute target, never add current position.
• --mousemove input = logical × output scale S; --getmouselocation returns logical.
• --windowmove -x/-y takes logical coordinates directly (NO × S).
• Multi-monitor: if window geometry is output-relative, add that output's position to WX,WY first.
• Use wlcctrl --outputs + --getdisplaygeometry for scale/position.
• Lock keys (CapsLock/NumLock) don't toggle reliably — use ydotool.

V10SP1 (xdotool) specifics

• No fractional scaling. Screen coordinates = window origin + relative offset. No scale factor anywhere.
• xdotool mousemove / windowmove take screen pixel coordinates directly (absolute).
• Screenshot via import (ImageMagick): import -window /tmp/win.png. Requires sudo apt install imagemagick.
• Window IDs are decimal integers (not UUIDs).
• xdotool getwindowgeometry --shell prints X=, Y=, WIDTH=, HEIGHT= for easy parsing.

Coordinate formula

V11 — multiply by scale (mousemove/clicks/drags)

Given window geometry (WX,WY), output scale S, target (RX,RY) in window screenshot:

logical_x = WX + RX
mousemove_x = round(logical_x * S)     # every --mousemove arg, start and end

V11 — windowmove: NO scale

windowmove -x = TX    # logical target, do NOT × S

V10SP1 — no scale at all (clicks/drags/windowmove all the same)

screen_x = WX + RX

The xdotool-window-click.sh / xdotool-drag.sh / xdotool-move-window.sh scripts do this for you.

Core commands

wlcctrl (V11)

wlcctrl --outputs; wlcctrl --getdisplaygeometry 
wlcctrl --list; wlcctrl --search ''; wlcctrl --getactivewindow
wlcctrl --getwindowname ; wlcctrl --getwindowgeometry 
wlcctrl --windowactivate 
wlcctrl --windowmove  -x  -y 
wlcctrl --windowsize  -w  -h 
wlcctrl --windowminimize|maximize|fullscreen  [-o ]
wlcctrl --windowmap 
wlcctrl --windowcapture  --path /tmp/w.png
wlcctrl --fullscreencapture --path /tmp/f.png [-o ]
wlcctrl --mousemove ,; wlcctrl --getmouselocation
wlcctrl --mousebutton 1; wlcctrl --mousepress 1 --mouserelease 1
wlcctrl --scroll -y 
wlcctrl --keystring 'text'; wlcctrl --key ctrl+s
wlcctrl --keydown ; wlcctrl --keyup 

xdotool (V10SP1)

xdotool search --name ''; xdotool search --onlyvisible --name ''
xdotool getactivewindow; xdotool getwindowname 
xdotool getwindowgeometry [--shell] 
xdotool windowactivate ; xdotool windowfocus 
xdotool windowmove   
xdotool windowsize   
xdotool getmouselocation; xdotool mousemove  
xdotool click ; xdotool mousedown ; xdotool mouseup 
xdotool type 'text'; xdotool key ctrl+s
xdotool keydown ; xdotool keyup 
import -window  /tmp/w.png        # screenshot (imagemagick)

When more detail is needed

Read references/details.md for keyboard aliases, multi-step test patterns, spinner controls, screenshots, multi-monitor handling, and X11 caveats.