Agent API

Overview

tde agent <verb> is the desktop's agent-facing API. It lets an AI agent — or any shell — list the desktop's windows, read exactly what each one is showing, inject keystrokes and mouse gestures, take screenshots, search scrollback, and drive the native apps through structured commands instead of screen-scraping.

It talks to a running desktop over an owner-only Unix socket. Each invocation sends one request line and prints the reply verbatim.

tde agent windows
tde agent send 3 "cargo test\n"
tde agent screen 3

Transport and limits

The client builds a request line from your arguments, connects to the desktop's control socket, writes exactly one newline-terminated frame, and reads the reply. Nothing here is HTTP and nothing listens on a port.

Pipelining is rejected: bytes after the first newline fail with invalid_frame, so a second command cannot be smuggled into a one-request connection.

Failures come back as a line beginning with error: followed by a JSON body carrying a stable code and a message. The prefix keeps older clients from mistaking a failure for success. When TDE is not running at all, the client says so explicitly rather than hanging; when a socket file exists but does not answer, it says that instead, because only the desktop owner may replace it.

Reading the desktop

screen is what the agent "sees": the flattened rendered canvas, not raw escape bytes. cells is the same content with per-cell attributes, for when styling matters.

desktop-screenshot is special on the client side. With no argument it prints the base64 payload; with a path it decodes it, verifies the PNG magic bytes and writes the file, printing ok: wrote <path>. Any other argument count is a usage error.

Sending input

tde agent send  ID TEXT
tde agent key   ID NAME
tde agent mouse ID <action> at COL ROW
tde agent mouse ID <action> on LABEL

send writes text into the window's PTY with escapes interpreted: \n, \r, \t, \e (escape) and \0, plus \\ for a literal backslash. An unrecognised escape is passed through unchanged.

key takes a name rather than bytes:

Names are case-insensitive. Anything else returns error: unknown key.

mouse actions are click, press, release, move, scrollup and scrolldown. The target is either at COL ROW — zero-based cell coordinates, comma or space separated — or on LABEL, which resolves a visible label to a cell. Gestures are only injected when the target app is actually in mouse reporting mode; otherwise the bytes would land as typed input, so TDE refuses.

Waiting for the screen to settle

tde agent wait 3 "Compiling" --timeout 15000

wait polls screen ID until TEXT appears, then prints the screen. The timeout is in milliseconds and defaults to 10000. Everything between the id and --timeout is joined into the text to look for, so it does not need quoting — but it is a plain substring, not a pattern.

Scrollback

tde agent scrollback        ID [LINES]
tde agent scrollback-search ID QUERY [LIMIT]
tde agent scrollback-export ID PATH [text|json]

scrollback returns scrolled-off history lines and defaults to 200. It reads both generic PTY windows and TDE Shell windows, so an agent is never blind to the terminal it is working in.

scrollback-search searches retained scrollback plus the visible primary content. LIMIT defaults to 100 and may not exceed 1000. scrollback-export writes the same content to a file — text (the default) or json. Exports are capped at 1 MiB, written atomically with owner-only permissions, and truncated from the oldest line when they would exceed the cap, with the truncation flagged in the payload.

Opening things

tde agent open-url https://example.com
tde agent run files

open-url opens an http or https URL in the native TDE Browser; anything else is rejected. It is the same bridge injected as $BROWSER into TDE-hosted terminal apps, which is how an OAuth flow inside a CLI tool ends up in TDE's browser rather than nowhere.

run opens a built-in TDE surface by its stable action id.

Web Share

tde agent web-share start local|tailnet|public
tde agent web-share status
tde agent web-share url
tde agent web-share stop

Three reach names, matching the deterministic CLI. status never returns the URL; url is the only command that does. Stopping an already-stopped share succeeds. See Share for the reach semantics and the loopback-versus-LAN caveat.

Driving the native apps

These verbs address a specific window and fail if that window is not the app they expect. They act on the app's model directly, which is why they are reliable in a way that reading the screen is not.

Sheets

tde agent sheets ID set A1 <value…>
tde agent sheets ID get A1
tde agent sheets ID dump
tde agent sheets ID new
tde agent sheets ID save <path>

set takes everything after the cell reference as the value, formulas included. dump returns tab-separated A1<TAB>value lines. save writes .xlsx and returns an error if a save is already in flight.

Docs

tde agent docs ID set <line> <text…>
tde agent docs ID append <text…>
tde agent docs ID get <line>
tde agent docs ID dump
tde agent docs ID new
tde agent docs ID save <path>

append returns the line number it wrote. dump returns the whole document.

API client

tde agent api ID method GET|POST|PUT|PATCH|DELETE|HEAD|OPTIONS
tde agent api ID url <url>
tde agent api ID body <json…>
tde agent api ID header <key> <value>
tde agent api ID clearheaders
tde agent api ID send
tde agent api ID get

send performs the request off the UI thread and returns the response as JSON — status, timing, headers and body. get returns the last response without re-sending. Methods are case-insensitive; an unknown one is an error.

Email

tde agent email ID accounts
tde agent email ID list
tde agent email ID read <message-id>
tde agent email ID draft <to> | <subject> | <markdown>

accounts and list return tab-separated lines. read returns the headers and the markdown body. draft takes three pipe-separated fields and returns the new draft's id.

Browser network

Debugging why a request failed is the one thing a screenshot cannot answer. These verbs act on the network layer of a TDE Browser window, so an agent reads the traffic itself.

tde agent browser ID capture start|stop|clear|status
tde agent browser ID packets [--filter <substring>] [--errors] [--limit <n>]
tde agent browser ID body <request-id>
tde agent browser ID har [--raw]

Capture is off until you start it — a browser that recorded everything by default would accumulate response bodies nobody asked for. packets lists what was captured; --errors narrows to failures and non-2xx responses. body returns one response body in full.

Replaying a captured request is how you test a fix without driving the UI back to the same state:

tde agent browser ID replay-preview <request-id> [overrides…]
tde agent browser ID replay <request-id> [overrides…]

Overrides are --method, --url, --header <k> <v>, --remove-header <k>, --body <json…>, --repeat <n>, --delay-ms <n> and --include-secrets. replay-preview shows the request that would be sent without sending it.

Breakpoints pause, fail or modify matching requests:

tde agent browser ID bp add <url-substring> [--stage request|response] [--action <action>]
tde agent browser ID bp list
tde agent browser ID bp remove <rule-id>
tde agent browser ID bp clear

Emulation makes the page believe it is somewhere, or something, else:

tde agent browser ID emulate throttle offline|slow-3g|fast-3g|wifi
tde agent browser ID emulate geo <lat> <lon> [accuracy]
tde agent browser ID emulate timezone <tz>   |  locale <lang>
tde agent browser ID emulate ua <string>     |  viewport <w> <h> [mobile|desktop]
tde agent browser ID emulate permissions <…> |  touch <on|off>  |  cpu <rate>
tde agent browser ID emulate status
tde agent browser ID emulate reset

Emulation persists on the window until reset, so a task that throttles to slow-3g and then reports a timing regression should reset first — otherwise the next agent inherits the handicap and measures the wrong thing.

Git review

tde agent git review add --task <task-id> --file <repo-relative-path> \
                         --line <n> [--side old|new] --text <comment>
tde agent git review list --task <task-id>

Unlike every other verb on this page, git review acts on the My Loop board directly — no running desktop and no open window required. add persists one inline diff review note against the task; list reads the task's notes back out. --side selects which side of the diff the line number refers to and defaults to the new side.

Quick reference