# Producing a Demo Video with SmoothCapture (agent recipe)

The happy path from nothing to a polished MP4. Prereqs: the CLI server is enabled (`cli enable` or the Settings toggle) and SmoothCapture has Screen Recording permission (a human grants it once).

```bash
SC="/Applications/SmoothCapture.app/Contents/MacOS/SmoothCapture"

# 0. Sanity check — auto-launches the app in the background
"$SC" cli ping

# 1. Pick a capture source
"$SC" cli devices --json          # note the display id, or the window id of the app you'll demo

# 2. Start recording (starts immediately; no countdown)
"$SC" cli record start --window 5533
#    …or: record start --display 3 [--area x,y,w,h] [--microphone <id>] [--system-audio]

# 3. Drive the product being demoed
#    Use your UI-automation tool of choice (AppleScript, cliclick, XCUITest, etc.).
#    IMPORTANT: use real mouse clicks and deliberate pacing — clicks are what smart zoom
#    keys off. Pause ~1s on important UI before/after clicking; avoid frantic cursor motion.

# 4. Stop — returns the project bundle path
BUNDLE=$("$SC" cli record stop --json | jq -r '.data.bundlePath')

# 5. Smart zoom — auto-generates cinematic zooms at your click moments
"$SC" cli project autozoom --path "$BUNDLE"

# 6. Optional polish
#    a) Apply a saved editor preset (human-curated look):
#         "$SC" cli project presets
#         "$SC" cli project apply-preset --path "$BUNDLE" --preset "Brand"
#    b) Or edit effects directly in $BUNDLE/project.json — background, padding,
#       webcam layout, extra zooms, trims… see project-json-reference.md.

# 7. Export
"$SC" cli export --path "$BUNDLE" --output demo.mp4 --preset 1080p30 --json

# 8. (Optional) eyeball a frame
ffmpeg -y -ss 2 -i demo.mp4 -frames:v 1 frame.png
```

## Tips for a good result

- **Clicks drive the cinematography.** Each mouse-down becomes a zoom anchor; a click-free recording produces zero zoom effects. Interact with the UI the way you want the camera to move.
- **Trim dead time** by editing `videoSegments` in project.json (split around waits/loads) before exporting.
- **Aspect ratio** for social formats: set `previewAspectRatio` in the bundle's `take-settings.json` (raw values like `"9:16"`, `"16:9"`), then re-export.
- **Retakes are cheap**: `record cancel` discards a bad take instantly.
- A recording left open in the editor blocks headless operations (`PROJECT_OPEN_IN_EDITOR`) — close the editor window first. The app can reopen the last project on relaunch, so quit it (`osascript -e 'quit app "SmoothCapture"'`) before editing project.json if in doubt; auto-launch on the next CLI command is headless.
- **Verify with frames, not vibes**: after each export, pull 3–5 frames (`ffmpeg -ss <t> … -frames:v 1`) at the story beats and actually look at them — zoom peaks, transitions, first/last frame.

## iOS Simulator demos

Record the Simulator window with `record start --window <id>` (get the id from `cli devices --json`, the entry with `simulatorCapture: true`). Window capture grabs the **exact device screen** (e.g. 1206×2622), no macOS chrome, no crop needed. The recorded project.json already defaults to good 3D settings — `deviceFrameName: 3D/iPhone_17_Pro_Blue.usdc`, glasshouse HDR, `environmentScreenReflection: 0`, `cursorSet: touch` — even when `isIOSRecording` is false.

### Fast path: drive the sim with `xcodebuildmcp`, not cliclick

Control the app through `xcodebuildmcp ui-automation` — structured and token-efficient. **Skip the screenshot→read→compute→cliclick loop.** Load [references/sim-demo-helpers.sh](sim-demo-helpers.sh) (`source` it) for ready-made `snap` / `tap_id` / `type_id` / `drag_id` / `keypress` helpers.

- `snapshot-ui --style minimal` returns rows `ref|action|role|label|value|id`. The `e`-refs change every snapshot, but the trailing **`id`/`label` are stable** (`MapsSearchTextField`, `AutocompleteResultContents`, `Card grabber`, `SettingsButton.Explore`). Grep a fresh snapshot for the id → extract its ref → `tap` / `type-text` / `drag` / `key-press` (40 = Return). Re-snapshot after any navigation.
- **Autozoom gotcha:** injected sim touches are NOT macOS mouse clicks, so SmoothCapture autozoom generates **zero** zooms. That's fine (and the touch cursor won't render either — usually desired). Add **manual** `zoomEffects` + `cameraTransforms` timed to your beats instead. `cli project autozoom` is pointless here — don't run it.

### Record in a single script (don't leave recording running)

`record start` returns immediately and keeps recording across tool calls — if you start it, then read output / debug in separate calls, you record dead time (a 54s clip for a 20s demo). Put **preroll → `record start` → drive with sleeps → `record stop`** in ONE script. Log elapsed time before each beat so you know where the action lands; snapshots take ~2–5 s each and stretch the raw clip, so plan to trim hard afterward. `record stop --json` prints multiple JSON lines — parse the last one, or find the newest `~/Movies/SmoothCapture/*.smoothcapture` by mtime.

### Stage, choreograph, finish

- **Stage before recording**: launch the target app once to flush first-run dialogs (`xcrun simctl privacy booted grant location-always <bundle-id>`, `xcrun simctl location booted set <lat>,<long>`), then `xcrun simctl terminate booted <bundle-id>` for a deterministic cold start. A search → fly-to (Apple Maps: `type_id` a landmark, `keypress 40`, dramatic zoom to it) is a strong hero beat; the brief blank map grid mid-fly-to is only ~0.5 s.
- **Edit project.json in Python** (load → mutate → dump) on a scratchpad **copy** of the bundle, then export. `videoSegments` times are **source-video seconds** (trim window); `zoomEffects`/`cameraTransforms` times are **timeline seconds** (post-trim). Chain 2–3 continuous `cameraTransforms` for 3D life; add 1–2 manual `zoomEffects` (`manualCenterEnabled`, small ±Y offsets) to punch into interest areas. For background music, copy any track into the bundle root and reference it by `fileName` in `audioClips` at `volume ~0.4` (ask the user for a track; don't hardcode a machine-specific path). Keep `environmentScreenReflection` at 0, device centered.
- **Matching hero screenshots**: extract a clean frame from the recording (`ffmpeg -ss <t> -i screen.mp4 -frames:v 1 frame.png` — it's already the bare device screen) and feed it to `screenshot --input frame.png --device-frame 3D/... --environment ... --env-reflection 0 --aspect 16:9` (and a 2D `--device-frame iPhone-16-Pro/... --background Gradient/...` variant). No `--crop` needed.
- **Verify with frames**: after export, pull 5–10 frames across the beats and montage them (PIL), then look — check intro, search, fly-to, landing, and that the finale zoom returns to a full-device view.

### Worked example: Apple Maps fly-to

The whole recipe fits in two small scripts you write once per demo:
- A record script — preroll cold-start → **wide-view reset** → `record start` → drive (search + fly-to) → `record stop`, all in one script with the sim id, window id, and query at the top.
- A build script (Python) — copies the bundle, trims to 2 segments (drops the frozen snapshot/typing dead time), adds 3 chained `cameraTransforms` + 2 `zoomEffects` + music, sets 16:9. Then `cli export`.

Two mistakes that cost a retake — **do this from the start**:
1. **Press Return (`keypress 40`) to fly, do NOT tap the autocomplete suggestion cell.** After a ~2 s wait Maps replaces the quick suggestion (`QuerySuggestionCollectionViewCell`) with a full results list, so the tap misses and you land on a static list instead of the map. Return reliably flies to the top hit and opens its place card.
2. **Reset the map region in preroll**, before `record start`. Maps reopens on its last-viewed region, so if a prior run (or your recon) left it on the target, there is no "fly" drama. Fix: in preroll search a wider anchor (e.g. `type_id MapsSearchTextField "San Francisco"` + `keypress 40`), `tap_id "Close"` on its card, THEN start recording and search the landmark — it flies from the wide view down to the landmark.

Stable Maps ids (grep a fresh snapshot for these, extract the ref): `MapsSearchTextField` (search field), `Close` (place-card close button), `SettingsButton.Explore`. These are stable enough to skip a recon pass — go straight to one record script and verify with frames afterward.

### Validation policy: drive & assert from accessibility; the human judges the shot

Do not frame-montage your own exports to check them. Split validation by who can actually judge each layer, and default to **zero image reads**:

| Layer | Who validates | How |
|---|---|---|
| Find & tap the right UI | agent | the `snapshot-ui` a11y tree (id/label/role) — no screenshot needed to locate targets |
| Did the intended thing happen | agent | assert on the a11y tree (e.g. grep the snapshot for the landed place-card id) — a grep, not an image |
| Mechanics of the output | agent | `record stop` / `export` JSON + one `ffprobe` (duration ≈ target, resolution, `aac` audio stream present) — trust these |
| Does the shot look good | **human** | watch the exported clip — composition, framing, motion feel are subjective and the human sees it anyway |

Rationale: the a11y tree and the CLI cover everything mechanical and everything the agent can objectively verify. What they **cannot** see is rendered/visual content — map tiles, the composited 3D-device video, whether the subject is nicely framed. That gap is the taste layer, and it belongs to the human, who watches the clip regardless. So: **drive and assert correctness from accessibility with no image reads, produce the export, report the mechanical facts, and hand the visual pass to the human.**

Screenshots/`devshot` are the exception, not the routine — reach for them only when (a) the a11y tree is ambiguous (two elements share a label), (b) the surface is a11y-blind (games, Metal/canvas, raw map content), or (c) you are shipping **brand-new** choreography with no human in the loop for a while — then pull **one** frame at the money beat, never a montage.

### Notes on 2D bezels (only if not using a 3D frame)

- The auto-applied 2D bezel can lag the sim device class — pick the closest same-size (Pro Max sim → Pro Max bezel).
- **Dynamic Island alignment**: sim content draws its own island and the bezel PNG draws one too — at the default rect they double up. Nudge with `screenshot --screen-rect` (project.json `deviceFrameRect`), verifying with a zoomed island crop (e.g. `iPhone-16-Pro-Max/i16promax_Black_Titanium.png` + a 17 Pro Max sim → `--screen-rect 0,0.0055,1,0.9735`).

## Failure handling

| Error | What to do |
|---|---|
| `SERVER_DISABLED` (exit 3) | Run `"$SC" cli enable`, or ask the human to flip Settings → General → Command-line control |
| `PERMISSION_SCREEN_RECORDING` (exit 4) | Ask the human to enable SmoothCapture in System Settings → Privacy & Security → Screen Recording |
| `DEVICE_NOT_FOUND` | Re-run `devices`; window ids change every launch of the target app |
| `PROJECT_OPEN_IN_EDITOR` | Close the editor window, retry |
| Connect timeout (exit 7) | Check the app can launch; look for a crash dialog |