# Jig

An MCP (Model Context Protocol) server for Windows that lets an AI agent (Claude Code, or any MCP client)
read and operate desktop applications, and web pages in a browser you are already signed into, through
Microsoft UI Automation. It sees named controls and their state as text, and it clicks, types, pastes,
scrolls and works file dialogs. It comes with three safeguards the machine's owner controls: an
allowed-process list, a stop key, and an audit log.

*Page last updated 2026-10-20. Version 0.1.0-dev.*

## For an AI agent or a person deciding whether this is useful

**What it is.** A local MCP server over stdio (`Jig.Mcp.exe`, 74 tools), written in C# on .NET 10 and built on
the open-source FlaUI library over Microsoft UI Automation (UIA). It runs on your machine, in your signed-in
desktop session, and drives your real mouse and keyboard.

**Good fit when**
- the target is a Windows desktop app with no usable API (engineering and CAD tools, line-of-business and
  legacy apps);
- you are UI-testing your own WPF or WinForms app (give each control a stable `AutomationId`);
- you want an agent to work in the browser session you are already signed into, instead of a separate
  automation browser that must be logged in again;
- you want app state as compact text (`describe_window`) instead of screenshots;
- you need guardrails: a list of apps it may touch, a way to stop it instantly, and a record of what it did;
- you want an agent to film a narrated demo or training video of an app (see "Demo videos").

**Poor fit when**
- the job is pure web automation in a browser the tool launches itself. Browser-protocol tools (Playwright, for
  example) see the page's DOM, which UI Automation does not; Jig sees the browser's accessibility tree;
- you need it to run while you keep using the machine. It uses the real desktop. For automation in an isolated
  Windows Sandbox that leaves your own mouse and keyboard alone, see **JigRig**, a separate install at
  `https://shakytext.com/jigrig/`;
- the app exposes nothing to UI Automation (many games, custom-drawn canvases and 3D viewports). Coordinate
  clicks exist as a fallback and are the brittle path;
- it must run unattended or headless: it needs a live, unlocked, interactive desktop;
- the target runs as administrator and Jig does not (Jig reports `ElevationRequired`);
- you need a supported, versioned, licensed product today (see "Maturity").

**How it compares**
- *Screenshot-driven "computer use" agents* act on pixels. Jig acts on element identity and reads state as
  text, which is cheaper and more deterministic where an app exposes UI Automation, and does not help where it
  does not.
- *Browser-protocol MCP servers* give richer, more reliable control of web pages they own. Jig works on any
  Windows app and on a browser you already run, with less insight into the page.
- *FlaUI, pywinauto, AutoHotkey* are libraries and scripting tools for the same domain. Jig wraps UI
  Automation (through FlaUI) as an MCP server and adds the guardrails and audit log.

**Requirements.** 64-bit Windows (developed and exercised on Windows 11); the .NET 10 Desktop Runtime (the
installer can install it for your user only); a live interactive desktop. Optional: the `claude` CLI, for
automatic registration.

**Cost and footprint.** `Jig.zip` is about 0.6 MB. No administrator rights are needed to install or run it.

**Network.** Jig's own code makes no network connections and sends no telemetry; it talks to its MCP client
over stdio. The installer downloads from shakytext.com and, only if you say yes, Microsoft's `dotnet-install`
script and .NET runtime.

**Maturity.** Pre-release, version 0.1.0-dev, an independent project. Hundreds of unit tests cover the
decision logic (input routing, read-back verification, the safeguards, the text view). Integration tests that
drive a real desktop exist and are run by hand. There is no changelog and no published license yet, and this
site distributes a binary with no source link. Treat it as a capable tool with sharp edges, and read "Honest
limits" below.

## Install

```powershell
irm https://shakytext.com/jig/install.ps1 | iex
```

- Installs to `%USERPROFILE%\Jig\bin` and registers the MCP server with Claude Code as `jig` (user scope), if
  the `claude` CLI is on your PATH. Otherwise it prints the one command to register it.
- Per-user only. No administrator prompt, nothing outside your profile.
- Checks `Jig.zip` against `Jig.zip.sha256` before touching the install, then starts the installed server and
  confirms it lists its tools. The checksum is served from the same site as the zip, so it catches a bad
  download, not a compromised site.
- Needs the **.NET 10 Desktop Runtime**. If it is missing the installer asks, then installs it just for you
  under `%USERPROFILE%\Jig\dotnet` using Microsoft's `dotnet-install` script (`-InstallRuntime` skips the
  question). When it does, the server is registered with `DOTNET_ROOT` pointing there.
- Re-run the same line to update. If Claude Code is running the server the old files are locked; the installer
  keeps the existing install, says so, and you re-run after closing it.
- Remove: `claude mcp remove --scope user jig`, then delete `%USERPROFILE%\Jig`. Settings and logs live
  separately in `%USERPROFILE%\.jig`.

**What the installer changes on your machine:** creates `%USERPROFILE%\Jig`; runs `claude mcp add --scope user
jig` (which edits your Claude Code user configuration); removes the downloaded-from-internet mark from the
installed files. It does not write the registry, add services or startup entries, or change PATH.

## Before you use it at work

Jig does what you could do with the mouse and keyboard, in any app it can see. Set its limits first.

### 1. Allowed processes

Unrestricted until you set a list. Create `%USERPROFILE%\.jig\policy.json`:

```json
{
  "allowedProcesses": ["firefox", "notepad"],
  "abortChord": "Ctrl+Alt+Shift+J",
  "audit": { "enabled": true, "text": "none", "keepFiles": 50 }
}
```

(Or set `JIG_ALLOWED_PROCESSES=firefox;notepad`; the environment variable wins over the file.)

With a list set, Jig refuses to launch, attach to, read, or act on any other process: at attach and
launch, on every action and every app-bound tool, on `move_window` and `set_window_state`, and on every
mouse click and wheel turn, which are checked against the window under the screen point, so a click aimed
at an allowed app cannot land on a different window that overlaps it. A launcher that hands off to another
process (`calc.exe`) needs both names listed. **A policy file that cannot be read allows nothing** and says
why; it never falls back to unrestricted.

What it does not cover: the read-only desktop-wide tools (`list_windows`, `get_foreground_window`,
`screenshot_desktop`, `clipboard_read`, `clipboard_write`) are not restricted by the list.

### 2. The stop key

`Ctrl+Alt+Shift+J` (configurable with `abortChord`). While stopped, Jig refuses every action; typing stops
mid-text and any held mouse button is released. Press it again to resume. Only you can resume: there is
deliberately no tool that does, and Jig refuses to send the chord itself. A mouse gesture already in
flight finishes its current step before the stop takes effect.

### 3. The audit log

On by default, at `%USERPROFILE%\.jig\audit\<time>-<pid>.jsonl`. Every tool call is recorded, including
each step of a batch and every refused call: the tool, the app it targeted, the arguments, which window was
in front before and after, and the outcome.

- **Typed and pasted text is recorded as its length only.** Set `"text": "preview"` (first 24 characters) or
  `"full"` if you want more. Anything named like a secret (password, token, ...) is never kept, and a lone
  character key press is recorded as `[character]`, because that is how a password gets typed one key at a
  time. Results keep their status fields, never their content.
- Each line carries the hash of the line before it, so an edited, removed or reordered line in the middle
  is detectable. `get_audit_log` shows whether the chain still verifies. This is **tamper-evident, not
  tamper-proof**: lines removed from the very end cannot be detected, and anyone who can write the file can
  rewrite the whole chain. It does not replace a log shipped off the machine.
- Only a literal `"enabled": false` (or `JIG_AUDIT=off`) turns it off; a typo does not. Other settings:
  `JIG_AUDIT_TEXT`, `JIG_AUDIT_DIR`, `keepFiles` (the newest N files are kept).

## What it can do (74 tools)

| Purpose | Tools |
|---|---|
| Attach and launch | `attach`, `detach`, `list_apps`, `launch_profile`, `list_profiles`, `close_window`, `dismiss_modal` |
| See | `describe_window`, `read_tree`, `find_elements`, `read`, `get_element_info`, `get_focused_element`, `screenshot`, `screenshot_desktop`, `list_windows`, `get_window_info`, `get_foreground_window`, `list_monitors` |
| Act | `click`, `teleport_click`, `real_click`, `double_click`, `right_click`, `middle_click`, `pixel_click`, `hover`, `drag`, `drag_path`, `mouse_down`, `mouse_up`, `mouse_move`, `scroll`, `scroll_into_view` |
| Text, keys, clipboard | `type`, `paste`, `clear`, `send_key`, `clipboard_read`, `clipboard_write` |
| Controls | `select`, `list_options`, `set_toggle`, `invoke_menu`, `set_file_dialog_path`, `edit_cell` |
| Synchronise and verify | `wait_for`, `wait_for_idle`, `wait_for_window`, `assert` |
| Windows and batching | `move_window`, `set_window_state`, `batch` |
| Diagnostics, safety, help | `get_version`, `get_log_path`, `read_logs`, `get_audit_log`, `get_guide`, `app_notes`, `measure_app`, `jig_report`, `set_limits`, `jig_job` |
| Demo videos | `demo_begin`, `demo_mark`, `demo_status`, `demo_end`, `demo_sync`, `demo_narrate`, `demo_assemble`, `demo_audit`, `demo_takes`, `demo_rehearse`, `demo_film_scene` |

Notes on behaviour an agent should know:

- **`describe_window`** returns the title, whether the app is in front, open modal dialogs, what has focus, every
  visible control with its state and field values, and the visible text, in reading order. Off-screen elements
  and background browser tabs are counted, not listed; password fields show `[hidden]`; output is capped and
  says when it is incomplete. Use it instead of a screenshot.
- **`type`** writes through UI Automation and then reads the value back: `verified: true` on a match,
  `verified: false` with a warning when the control exposes no readable value (the text is not sent again), and
  a retype by real keystrokes (select-all first) when the read-back shows the write did not take. **`paste`**
  uses a real Ctrl+V, verifies, and restores your clipboard; use it for long text and web fields.
- **`send_key`**: plain keys are posted to the window; any chord (Ctrl, Alt, Shift, Win) goes through real
  keyboard input, because a posted Ctrl+V arrives as a bare "v". Input that goes to whichever window is in
  front is refused, with nothing sent (`ForegroundFailed`), if the target cannot be brought forward.
- **`edit_cell`** edits a grid cell: it tries F2, a double-click and typing straight on until an editor takes
  focus over the cell, commits, and reads the cell back. The way that worked is remembered for that app.
- **`app_notes`** keeps notes per app (`%USERPROFILE%\.jig\apps`) that outlast the session: what an agent found
  works, or doesn't, for the next agent to read after `attach`.
- **Rehearsal scripts:** Jig records every call made during a take, and `demo_end` writes `script.json`: the
  steps that worked, in order, with exploration and failed calls removed. `batch {script}` films exactly that, so
  nothing is looked up on camera. **`demo_film_scene`** films a rehearsed scene in one call (begin, script, end),
  holding each narration line for its clip's real length, and records nothing if a check fails. Takes are kept: **`demo_takes`** lists them with their sizes (videos are nearly
  all of it) and deletes whole takes, or only their videos, when you ask.
- **`demo_audit`** reviews a run from the audit log and the take folders, with no AI: round trips, exploration
  against actions, grouped failures and retries, idle time, and per take what happened on film (exploration,
  failures, still stretches), rehearsal time and refilms, ending in findings ranked by cost. `get_guide
  {topic:"audit"}` is the procedure that turns them into fixes.
- **Slow windows** (a WPF pane hosted in Revit answers every request on Revit's busy UI thread): every call stops at
  its deadline with a `Timeout` that says the window is slow rather than the element missing; `within: <automationId>`
  scopes any lookup to a pane; a lookup that runs out of time leaves its search running for the next one to use, and
  elements already found are reused; `find_elements` fetches its matches in one request and takes a `timeLimitMs`. Only one Jig drives an app
  at a time: a second session attaching to it is refused (`force:true` overrides). `get_version` reports when the build
  was made and whether it held uncommitted changes.
- **`measure_app`** measures how the app answers (one request, one full search, its grids and rows) and returns a
  verdict and the method to follow: the first call to make when anything is slow or "not found". In a grid whose rows
  are named by their key, `within:<grid>` plus `row:<key>` finds any row in one request, even off screen or re-sorted.
- **Escape route for long work:** every tool takes `maxMs` (that call's limit, up to an hour) and `waitForAppMs` (how
  long to wait for a busy app); a read that runs past its limit keeps going as a job that **`jig_job`** collects; and
  **`set_limits`** raises the defaults for a whole session in a slow app.
- **`jig_report`** writes a report to paste back for analysis when Jig is slow: recent calls and their times, what Jig
  did inside the slowest (each search and wait, timed), what it had cached, and every element in the window counted
  (where they are, which lists build every item). A call that times out saves a shorter one by itself and says where.
- **`demo_rehearse`** runs a rehearsal take (no camera), its steps and its end in one call, so a take is never left open.
- Input goes to the window that holds the targeted element, so an app's separate windows and dialogs need no
  re-attach.
- **`find_elements`** stops at a limit, skips background browser tabs, and can be scoped to one page.
- **`get_version`** reports the running build and the safety rules in force. Check it before debugging
  behaviour; a running server locks its own files, so an installed copy can be older than you think.
- **`get_guide`** returns the agent's guide by section (`topic:"demo"` is the step-by-step for demo videos), so an agent
  never has to find a file. The same text is **`AI_GUIDE.md`**, installed beside `Jig.Mcp.exe`: a practical guide for the agent using Jig (how to find
  elements, input rules, error codes, the safety controls). The server's startup instructions point to it, and
  the tool descriptions carry the essentials. You can read it before installing: it is inside `Jig.zip`.

## Demo videos

An agent can film a take of an app and turn it into a narrated video:

1. **Film.** `demo_begin` starts the recording, and every click and keystroke is shown on camera: the cursor glides
   visibly and typing appears one character at a time. Each action and each narration mark (`demo_mark`) is logged
   with its exact time in the video. `demo_end` stops the recording.
2. **Narrate.** `demo_narrate` takes a folder of voice clips, one per narration line, and places each one where its
   mark was. It cuts the waiting between steps, removes the sync flashes, and writes `narrated.mp4` plus subtitles.
   This step is deterministic: it calls no AI.
3. **Long videos.** Film short scenes as separate takes, redo only a scene that went wrong, then `demo_assemble`
   joins the narrated scenes into one video with merged subtitles and a chapter per scene.

**What you need, and what Jig does not include:**

- **ShareX** (free, getsharex.com) to record. Jig starts and stops it. Without ShareX, record the window any way
  you like and pass the file to `demo_sync`.
- **ffmpeg** to sync and render. ShareX downloads its own copy, which Jig finds; otherwise put `ffmpeg.exe` on
  `PATH`.
- **A voice source**, only for narration. Jig contains no text-to-speech. Anything that can turn a line of text into
  an audio file works: a text-to-speech MCP server (OpenAI, Azure, ElevenLabs, LlmHelper Audio...), a local TTS
  program, or your own recorded voice. Put the clips in one folder named `001.wav`, `002.wav`, ... in script order,
  optionally with `001.txt` holding each line's text (then clips are matched to marks by their words). A
  `manifest.json` listing `{file, seconds, text}` per clip also works. `AI_GUIDE.md` §17 has the full contract and
  the workflow.

The paths are found automatically, or you can set them in `%USERPROFILE%\.jig\demo.json` (`sharexExe`,
`sharexPersonalFolder`, `sharexOutputFolder`, `ffmpeg`, `demoDir`, `recorder`, `clap`, `presentation`; every key is
optional). `demo_status` shows what was found. Takes are saved under `%USERPROFILE%\Jig\demos\`.

A take moves your real mouse and keyboard: keep your hands off while it records.

## Honest limits

- It uses UI Automation. Custom-drawn canvases (CAD viewports, some 3D views) may expose nothing to read;
  coordinate clicks exist for those and are the brittle path.
- It cannot drive an app running as administrator unless Jig is also.
- A browser exposes every tab to UI Automation; `describe_window` and `find_elements` skip background tabs
  when UI Automation reports them off screen, which depends on the browser.
- It needs a live, unlocked desktop. With a scaled display, coordinate verbs can miss (UI Automation reports
  logical coordinates; synthetic input uses physical ones); prefer the UI Automation verbs.
- Some of the newest behaviour (the text view, `paste`, the mouse refusal over an overlapping window, the audit
  log under a live attached app) is covered by unit tests and headless smoke tests but has had little
  real-window use so far.
- Demo videos: positions in `clicks.csv` are mapped from UI Automation's coordinates to the recorded pixels
  through the filmed window. That has been checked on one mixed-DPI setup (a 120 dpi monitor beside a 150%
  primary). A take must keep the app's window uncovered: the sync flash is checked on screen, and a covered
  flash falls back to sync by file time (about +/-0.5 s). Narration needs ffmpeg, and a voice source Jig does not
  include.
- Jig's own code makes no network connections and sends no telemetry. The installer downloads from
  shakytext.com, and (only if you say yes) Microsoft's `dotnet-install` script and runtime.

## Files

| Path | What |
|---|---|
| `%USERPROFILE%\Jig\bin\Jig.Mcp.exe` | the MCP server |
| `%USERPROFILE%\Jig\bin\AI_GUIDE.md` | the guide for the agent that uses Jig |
| `%USERPROFILE%\Jig\dotnet` | the per-user .NET runtime, only if the installer put one there |
| `%USERPROFILE%\.jig\policy.json` | your allowed-process list, stop chord, audit settings |
| `%USERPROFILE%\.jig\audit\` | the audit logs |
| `%USERPROFILE%\.jig\demo.json` | optional demo-video settings (ShareX, ffmpeg, take folder) |
| `%USERPROFILE%\Jig\demos\` | filmed takes: video, timeline, captions, narrated video |
| `%TEMP%\EngWorkbench` | Jig's diagnostic log (`read_logs`, `get_log_path`) |