chore: Run pnpm format:fix.

This commit is contained in:
cpojer
2026-01-31 21:13:13 +09:00
parent dcc2de15a6
commit 8cab78abbc
624 changed files with 10729 additions and 7514 deletions

View File

@@ -26,23 +26,28 @@ The macOS apps **Install CLI** button runs the same flow via npm/pnpm (bun no
## Launchd (Gateway as LaunchAgent)
Label:
- `bot.molt.gateway` (or `bot.molt.<profile>`; legacy `com.openclaw.*` may remain)
Plist location (peruser):
- `~/Library/LaunchAgents/bot.molt.gateway.plist`
(or `~/Library/LaunchAgents/bot.molt.<profile>.plist`)
Manager:
- The macOS app owns LaunchAgent install/update in Local mode.
- The CLI can also install it: `openclaw gateway install`.
Behavior:
- “OpenClaw Active” enables/disables the LaunchAgent.
- App quit does **not** stop the gateway (launchd keeps it alive).
- If a Gateway is already running on the configured port, the app attaches to
it instead of starting a new one.
Logging:
- launchd stdout/err: `/tmp/openclaw/openclaw-gateway.log`
## Version compatibility

View File

@@ -5,6 +5,7 @@ read_when:
- Adding agent controls for visual workspace
- Debugging WKWebView canvas loads
---
# Canvas (macOS app)
The macOS app embeds an agentcontrolled **Canvas panel** using `WKWebView`. It
@@ -22,6 +23,7 @@ The Canvas panel serves those files via a **custom URL scheme**:
- `openclaw-canvas://<session>/<path>`
Examples:
- `openclaw-canvas://main/``<canvasRoot>/main/index.html`
- `openclaw-canvas://main/assets/app.css``<canvasRoot>/main/assets/app.css`
- `openclaw-canvas://main/widgets/todo/``<canvasRoot>/main/widgets/todo/index.html`
@@ -57,6 +59,7 @@ openclaw nodes canvas snapshot --node <id>
```
Notes:
- `canvas.navigate` accepts **local canvas paths**, `http(s)` URLs, and `file://` URLs.
- If you pass `"/"`, the Canvas shows the local scaffold or `index.html`.

View File

@@ -3,6 +3,7 @@ summary: "Gateway lifecycle on macOS (launchd)"
read_when:
- Integrating the mac app with the gateway lifecycle
---
# Gateway lifecycle on macOS
The macOS app **manages the Gateway via launchd** by default and does not spawn

View File

@@ -3,6 +3,7 @@ summary: "Setup guide for developers working on the OpenClaw macOS app"
read_when:
- Setting up the macOS development environment
---
# macOS Developer Setup
This guide covers the necessary steps to build and run the OpenClaw macOS application from source.
@@ -30,7 +31,7 @@ To build the macOS app and package it into `dist/OpenClaw.app`, run:
./scripts/package-mac-app.sh
```
If you don't have an Apple Developer ID certificate, the script will automatically use **ad-hoc signing** (`-`).
If you don't have an Apple Developer ID certificate, the script will automatically use **ad-hoc signing** (`-`).
For dev run modes, signing flags, and Team ID troubleshooting, see the macOS app README:
https://github.com/openclaw/openclaw/blob/main/apps/macos/README.md
@@ -42,11 +43,13 @@ https://github.com/openclaw/openclaw/blob/main/apps/macos/README.md
The macOS app expects a global `openclaw` CLI install to manage background tasks.
**To install it (recommended):**
1. Open the OpenClaw app.
2. Go to the **General** settings tab.
3. Click **"Install CLI"**.
Alternatively, install it manually:
```bash
npm install -g openclaw@<version>
```
@@ -54,13 +57,16 @@ npm install -g openclaw@<version>
## Troubleshooting
### Build Fails: Toolchain or SDK Mismatch
The macOS app build expects the latest macOS SDK and Swift 6.2 toolchain.
**System dependencies (required):**
- **Latest macOS version available in Software Update** (required by Xcode 26.2 SDKs)
- **Xcode 26.2** (Swift 6.2 toolchain)
**Checks:**
```bash
xcodebuild -version
xcrun swift --version
@@ -69,9 +75,11 @@ xcrun swift --version
If versions dont match, update macOS/Xcode and re-run the build.
### App Crashes on Permission Grant
If the app crashes when you try to allow **Speech Recognition** or **Microphone** access, it may be due to a corrupted TCC cache or signature mismatch.
**Fix:**
1. Reset the TCC permissions:
```bash
tccutil reset All bot.molt.mac.debug
@@ -79,6 +87,7 @@ If the app crashes when you try to allow **Speech Recognition** or **Microphone*
2. If that fails, change the `BUNDLE_ID` temporarily in [`scripts/package-mac-app.sh`](https://github.com/openclaw/openclaw/blob/main/scripts/package-mac-app.sh) to force a "clean slate" from macOS.
### Gateway "Starting..." indefinitely
If the gateway status stays on "Starting...", check if a zombie process is holding the port:
```bash
@@ -88,4 +97,5 @@ openclaw gateway stop
# If youre not using a LaunchAgent (dev mode / manual runs), find the listener:
lsof -nP -iTCP:18789 -sTCP:LISTEN
```
If a manual run is holding the port, stop that process (Ctrl+C). As a last resort, kill the PID you found above.

View File

@@ -3,11 +3,13 @@ summary: "How the macOS app reports gateway/Baileys health states"
read_when:
- Debugging mac app health indicators
---
# Health Checks on macOS
How to see whether the linked channel is healthy from the menu bar app.
## Menu bar
- Status dot now reflects Baileys health:
- Green: linked + socket opened recently.
- Orange: connecting/retrying.
@@ -16,13 +18,16 @@ How to see whether the linked channel is healthy from the menu bar app.
- "Run Health Check" menu item triggers an on-demand probe.
## Settings
- General tab gains a Health card showing: linked auth age, session-store path/count, last check time, last error/status code, and buttons for Run Health Check / Reveal Logs.
- Uses a cached snapshot so the UI loads instantly and falls back gracefully when offline.
- **Channels tab** surfaces channel status + controls for WhatsApp/Telegram (login QR, logout, probe, last disconnect/error).
## How the probe works
- App runs `openclaw health --json` via `ShellExecutor` every ~60s and on demand. The probe loads creds and reports status without sending messages.
- Cache the last good snapshot and the last error separately to avoid flicker; show the timestamp of each.
## When in doubt
- You can still use the CLI flow in [Gateway health](/gateway/health) (`openclaw status`, `openclaw status --deep`, `openclaw health --json`) and tail `/tmp/openclaw/openclaw-*.log` for `web-heartbeat` / `web-reconnect`.

View File

@@ -3,6 +3,7 @@ summary: "Menu bar icon states and animations for OpenClaw on macOS"
read_when:
- Changing menu bar icon behavior
---
# Menu Bar Icon States
Author: steipete · Updated: 2025-12-06 · Scope: macOS app (`apps/macos`)
@@ -13,14 +14,17 @@ Author: steipete · Updated: 2025-12-06 · Scope: macOS app (`apps/macos`)
- **Working (agent running):** `AppState.isWorking=true` drives a “tail/leg scurry” micro-motion: faster leg wiggle and slight offset while work is in-flight. Currently toggled around WebChat agent runs; add the same toggle around other long tasks when you wire them.
Wiring points
- Voice wake: runtime/tester call `AppState.triggerVoiceEars(ttl: nil)` on trigger and `stopVoiceEars()` after 1s of silence to match the capture window.
- Agent activity: set `AppStateStore.shared.setWorking(true/false)` around work spans (already done in WebChat agent call). Keep spans short and reset in `defer` blocks to avoid stuck animations.
Shapes & sizes
- Base icon drawn in `CritterIconRenderer.makeIcon(blink:legWiggle:earWiggle:earScale:earHoles:)`.
- Ear scale defaults to `1.0`; voice boost sets `earScale=1.9` and toggles `earHoles=true` without changing overall frame (18×18pt template image rendered into a 36×36px Retina backing store).
- Scurry uses leg wiggle up to ~1.0 with a small horizontal jiggle; its additive to any existing idle wiggle.
Behavioral notes
- No external CLI/broker toggle for ears/working; keep it internal to the apps own signals to avoid accidental flapping.
- Keep TTLs short (&lt;10s) so the icon returns to baseline quickly if a job hangs.

View File

@@ -4,9 +4,11 @@ read_when:
- Capturing macOS logs or investigating private data logging
- Debugging voice wake/session lifecycle issues
---
# Logging (macOS)
## Rolling diagnostics file log (Debug pane)
OpenClaw routes macOS app logs through swift-log (unified logging by default) and can write a local, rotating file log to disk when you need a durable capture.
- Verbosity: **Debug pane → Logs → App logging → Verbosity**
@@ -15,6 +17,7 @@ OpenClaw routes macOS app logs through swift-log (unified logging by default) an
- Clear: **Debug pane → Logs → App logging → “Clear”**
Notes:
- This is **off by default**. Enable only while actively debugging.
- Treat the file as sensitive; dont share it without review.
@@ -23,6 +26,7 @@ Notes:
Unified logging redacts most payloads unless a subsystem opts into `privacy -off`. Per Peter's write-up on macOS [logging privacy shenanigans](https://steipete.me/posts/2025/logging-privacy-shenanigans) (2025) this is controlled by a plist in `/Library/Preferences/Logging/Subsystems/` keyed by the subsystem name. Only new log entries pick up the flag, so enable it before reproducing an issue.
## Enable for OpenClaw (`bot.molt`)
- Write the plist to a temp file first, then install it atomically as root:
```bash
@@ -46,6 +50,7 @@ sudo install -m 644 -o root -g wheel /tmp/bot.molt.plist /Library/Preferences/Lo
- View the richer output with the existing helper, e.g. `./scripts/clawlog.sh --category WebChat --last 5m`.
## Disable after debugging
- Remove the override: `sudo rm /Library/Preferences/Logging/Subsystems/bot.molt.plist`.
- Optionally run `sudo log config --reload` to force logd to drop the override immediately.
- Remember this surface can include phone numbers and message bodies; keep the plist in place only while you actively need the extra detail.

View File

@@ -3,15 +3,18 @@ summary: "Menu bar status logic and what is surfaced to users"
read_when:
- Tweaking mac menu UI or status logic
---
# Menu Bar Status Logic
## What is shown
- We surface the current agent work state in the menu bar icon and in the first status row of the menu.
- Health status is hidden while work is active; it returns when all sessions are idle.
- The “Nodes” block in the menu lists **devices** only (paired nodes via `node.list`), not client/presence entries.
- A “Usage” section appears under Context when provider usage snapshots are available.
## State model
- Sessions: events arrive with `runId` (per-run) plus `sessionKey` in the payload. The “main” session is the key `main`; if absent, we fall back to the most recently updated session.
- Priority: main always wins. If main is active, its state is shown immediately. If main is idle, the most recently active nonmain session is shown. We do not flipflop midactivity; we only switch when the current session goes idle or main becomes active.
- Activity kinds:
@@ -19,12 +22,14 @@ read_when:
- `tool`: `phase: start|result` with `toolName` and `meta/args`.
## IconState enum (Swift)
- `idle`
- `workingMain(ActivityKind)`
- `workingOther(ActivityKind)`
- `overridden(ActivityKind)` (debug override)
### ActivityKind → glyph
- `exec` → 💻
- `read` → 📄
- `write` → ✍️
@@ -33,17 +38,20 @@ read_when:
- default → 🛠️
### Visual mapping
- `idle`: normal critter.
- `workingMain`: badge with glyph, full tint, leg “working” animation.
- `workingOther`: badge with glyph, muted tint, no scurry.
- `overridden`: uses the chosen glyph/tint regardless of activity.
## Status row text (menu)
- While work is active: `<Session role> · <activity label>`
- Examples: `Main · exec: pnpm test`, `Other · read: apps/macos/Sources/OpenClaw/AppState.swift`.
- When idle: falls back to the health summary.
## Event ingestion
- Source: controlchannel `agent` events (`ControlChannel.handleAgentEvent`).
- Parsed fields:
- `stream: "job"` with `data.state` for start/stop.
@@ -55,6 +63,7 @@ read_when:
- fallback: tool name.
## Debug override
- Settings ▸ Debug ▸ “Icon override” picker:
- `System (auto)` (default)
- `Working: main` (per tool kind)
@@ -63,6 +72,7 @@ read_when:
- Stored via `@AppStorage("iconOverride")`; mapped to `IconState.overridden`.
## Testing checklist
- Trigger main session job: verify icon switches immediately and status row shows main label.
- Trigger nonmain session job while main idle: icon/status shows nonmain; stays stable until it finishes.
- Start main while other active: icon flips to main instantly.

View File

@@ -5,6 +5,7 @@ read_when:
- Integrating Peekaboo via Swift Package Manager
- Changing PeekabooBridge protocol/paths
---
# Peekaboo Bridge (macOS UI automation)
OpenClaw can host **PeekabooBridge** as a local, permissionaware UI automation
@@ -20,6 +21,7 @@ macOS apps TCC permissions.
## Enable the bridge
In the macOS app:
- Settings → **Enable Peekaboo Bridge**
When enabled, OpenClaw starts a local UNIX socket server. If disabled, the host

View File

@@ -5,6 +5,7 @@ read_when:
- Packaging or signing the macOS app
- Changing bundle IDs or app install paths
---
# macOS permissions (TCC)
macOS permission grants are fragile. TCC associates a permission grant with the
@@ -12,6 +13,7 @@ app's code signature, bundle identifier, and on-disk path. If any of those chang
macOS treats the app as new and may drop or hide prompts.
## Requirements for stable permissions
- Same path: run the app from a fixed location (for OpenClaw, `dist/OpenClaw.app`).
- Same bundle identifier: changing the bundle ID creates a new permission identity.
- Signed app: unsigned or ad-hoc signed builds do not persist permissions.
@@ -22,6 +24,7 @@ Ad-hoc signatures generate a new identity every build. macOS will forget previou
grants, and prompts can disappear entirely until the stale entries are cleared.
## Recovery checklist when prompts disappear
1. Quit the app.
2. Remove the app entry in System Settings -> Privacy & Security.
3. Relaunch the app from the same path and re-grant permissions.

View File

@@ -10,6 +10,7 @@ read_when:
This app now ships Sparkle auto-updates. Release builds must be Developer IDsigned, zipped, and published with a signed appcast entry.
## Prereqs
- Developer ID Application cert installed (example: `Developer ID Application: <Developer Name> (<TEAMID>)`).
- Sparkle private key path set in the environment as `SPARKLE_PRIVATE_KEY_FILE` (path to your Sparkle ed25519 private key; public key baked into Info.plist). If it is missing, check `~/.profile`.
- Notary credentials (keychain profile or API key) for `xcrun notarytool` if you want Gatekeeper-safe DMG/zip distribution.
@@ -21,7 +22,9 @@ This app now ships Sparkle auto-updates. Release builds must be Developer IDs
- Sparkle tools are fetched automatically via SwiftPM at `apps/macos/.build/artifacts/sparkle/Sparkle/bin/` (`sign_update`, `generate_appcast`, etc.).
## Build & package
Notes:
- `APP_BUILD` maps to `CFBundleVersion`/`sparkle:version`; keep it numeric + monotonic (no `-beta`), or Sparkle compares it as equal.
- Defaults to the current architecture (`$(uname -m)`). For release/universal builds, set `BUILD_ARCHS="arm64 x86_64"` (or `BUILD_ARCHS=all`).
- Use `scripts/package-mac-dist.sh` for release artifacts (zip + DMG + notarization). Use `scripts/package-mac-app.sh` for local/dev packaging.
@@ -59,14 +62,18 @@ ditto -c -k --keepParent apps/macos/.build/release/OpenClaw.app.dSYM dist/OpenCl
```
## Appcast entry
Use the release note generator so Sparkle renders formatted HTML notes:
```bash
SPARKLE_PRIVATE_KEY_FILE=/path/to/ed25519-private-key scripts/make_appcast.sh dist/OpenClaw-2026.1.27-beta.1.zip https://raw.githubusercontent.com/openclaw/openclaw/main/appcast.xml
```
Generates HTML release notes from `CHANGELOG.md` (via [`scripts/changelog-to-html.sh`](https://github.com/openclaw/openclaw/blob/main/scripts/changelog-to-html.sh)) and embeds them in the appcast entry.
Commit the updated `appcast.xml` alongside the release assets (zip + dSYM) when publishing.
## Publish & verify
- Upload `OpenClaw-2026.1.27-beta.1.zip` (and `OpenClaw-2026.1.27-beta.1.dSYM.zip`) to the GitHub release for tag `v2026.1.27-beta.1`.
- Ensure the raw appcast URL matches the baked feed: `https://raw.githubusercontent.com/openclaw/openclaw/main/appcast.xml`.
- Sanity checks:

View File

@@ -3,29 +3,34 @@ summary: "macOS app flow for controlling a remote OpenClaw gateway over SSH"
read_when:
- Setting up or debugging remote mac control
---
# Remote OpenClaw (macOS ⇄ remote host)
This flow lets the macOS app act as a full remote control for a OpenClaw gateway running on another host (desktop/server). Its the apps **Remote over SSH** (remote run) feature. All features—health checks, Voice Wake forwarding, and Web Chat—reuse the same remote SSH configuration from *Settings → General*.
This flow lets the macOS app act as a full remote control for a OpenClaw gateway running on another host (desktop/server). Its the apps **Remote over SSH** (remote run) feature. All features—health checks, Voice Wake forwarding, and Web Chat—reuse the same remote SSH configuration from _Settings → General_.
## Modes
- **Local (this Mac)**: Everything runs on the laptop. No SSH involved.
- **Remote over SSH (default)**: OpenClaw commands are executed on the remote host. The mac app opens an SSH connection with `-o BatchMode` plus your chosen identity/key and a local port-forward.
- **Remote direct (ws/wss)**: No SSH tunnel. The mac app connects to the gateway URL directly (for example, via Tailscale Serve or a public HTTPS reverse proxy).
## Remote transports
Remote mode supports two transports:
- **SSH tunnel** (default): Uses `ssh -N -L ...` to forward the gateway port to localhost. The gateway will see the nodes IP as `127.0.0.1` because the tunnel is loopback.
- **Direct (ws/wss)**: Connects straight to the gateway URL. The gateway sees the real client IP.
## Prereqs on the remote host
1) Install Node + pnpm and build/install the OpenClaw CLI (`pnpm install && pnpm build && pnpm link --global`).
2) Ensure `openclaw` is on PATH for non-interactive shells (symlink into `/usr/local/bin` or `/opt/homebrew/bin` if needed).
3) Open SSH with key auth. We recommend **Tailscale** IPs for stable reachability off-LAN.
1. Install Node + pnpm and build/install the OpenClaw CLI (`pnpm install && pnpm build && pnpm link --global`).
2. Ensure `openclaw` is on PATH for non-interactive shells (symlink into `/usr/local/bin` or `/opt/homebrew/bin` if needed).
3. Open SSH with key auth. We recommend **Tailscale** IPs for stable reachability off-LAN.
## macOS app setup
1) Open *Settings → General*.
2) Under **OpenClaw runs**, pick **Remote over SSH** and set:
1. Open _Settings → General_.
2. Under **OpenClaw runs**, pick **Remote over SSH** and set:
- **Transport**: **SSH tunnel** or **Direct (ws/wss)**.
- **SSH target**: `user@host` (optional `:port`).
- If the gateway is on the same LAN and advertises Bonjour, pick it from the discovered list to auto-fill this field.
@@ -33,28 +38,33 @@ Remote mode supports two transports:
- **Identity file** (advanced): path to your key.
- **Project root** (advanced): remote checkout path used for commands.
- **CLI path** (advanced): optional path to a runnable `openclaw` entrypoint/binary (auto-filled when advertised).
3) Hit **Test remote**. Success indicates the remote `openclaw status --json` runs correctly. Failures usually mean PATH/CLI issues; exit 127 means the CLI isnt found remotely.
4) Health checks and Web Chat will now run through this SSH tunnel automatically.
3. Hit **Test remote**. Success indicates the remote `openclaw status --json` runs correctly. Failures usually mean PATH/CLI issues; exit 127 means the CLI isnt found remotely.
4. Health checks and Web Chat will now run through this SSH tunnel automatically.
## Web Chat
- **SSH tunnel**: Web Chat connects to the gateway over the forwarded WebSocket control port (default 18789).
- **Direct (ws/wss)**: Web Chat connects straight to the configured gateway URL.
- There is no separate WebChat HTTP server anymore.
## Permissions
- The remote host needs the same TCC approvals as local (Automation, Accessibility, Screen Recording, Microphone, Speech Recognition, Notifications). Run onboarding on that machine to grant them once.
- Nodes advertise their permission state via `node.list` / `node.describe` so agents know whats available.
## Security notes
- Prefer loopback binds on the remote host and connect via SSH or Tailscale.
- If you bind the Gateway to a non-loopback interface, require token/password auth.
- See [Security](/gateway/security) and [Tailscale](/gateway/tailscale).
## WhatsApp login flow (remote)
- Run `openclaw channels login --verbose` **on the remote host**. Scan the QR with WhatsApp on your phone.
- Re-run login on that host if auth expires. Health check will surface link problems.
## Troubleshooting
- **exit 127 / not found**: `openclaw` isnt on PATH for non-login shells. Add it to `/etc/paths`, your shell rc, or symlink into `/usr/local/bin`/`/opt/homebrew/bin`.
- **Health probe failed**: check SSH reachability, PATH, and that Baileys is logged in (`openclaw status --json`).
- **Web Chat stuck**: confirm the gateway is running on the remote host and the forwarded port matches the gateway WS port; the UI requires a healthy WS connection.
@@ -62,6 +72,7 @@ Remote mode supports two transports:
- **Voice Wake**: trigger phrases are forwarded automatically in remote mode; no separate forwarder is needed.
## Notification sounds
Pick sounds per notification from scripts with `openclaw` and `node.invoke`, e.g.:
```bash

View File

@@ -3,6 +3,7 @@ summary: "Signing steps for macOS debug builds generated by packaging scripts"
read_when:
- Building or signing mac debug builds
---
# mac signing (debug builds)
This app is usually built from [`scripts/package-mac-app.sh`](https://github.com/openclaw/openclaw/blob/main/scripts/package-mac-app.sh), which now:
@@ -28,11 +29,13 @@ DISABLE_LIBRARY_VALIDATION=1 scripts/package-mac-app.sh # dev-only Sparkle Tea
```
### Ad-hoc Signing Note
When signing with `SIGN_IDENTITY="-"` (ad-hoc), the script automatically disables the **Hardened Runtime** (`--options runtime`). This is necessary to prevent crashes when the app attempts to load embedded frameworks (like Sparkle) that do not share the same Team ID. Ad-hoc signatures also break TCC permission persistence; see [macOS permissions](/platforms/mac/permissions) for recovery steps.
## Build metadata for About
`package-mac-app.sh` stamps the bundle with:
- `OpenClawBuildTimestamp`: ISO8601 UTC at package time
- `OpenClawGitCommit`: short git hash (or `unknown` if unavailable)
@@ -40,4 +43,4 @@ The About tab reads these keys to show version, build date, git commit, and whet
## Why
TCC permissions are tied to the bundle identifier *and* code signature. Unsigned debug builds with changing UUIDs were causing macOS to forget grants after each rebuild. Signing the binaries (adhoc by default) and keeping a fixed bundle id/path (`dist/OpenClaw.app`) preserves the grants between builds, matching the VibeTunnel approach.
TCC permissions are tied to the bundle identifier _and_ code signature. Unsigned debug builds with changing UUIDs were causing macOS to forget grants after each rebuild. Signing the binaries (adhoc by default) and keeping a fixed bundle id/path (`dist/OpenClaw.app`) preserves the grants between builds, matching the VibeTunnel approach.

View File

@@ -4,24 +4,29 @@ read_when:
- Updating the macOS Skills settings UI
- Changing skills gating or install behavior
---
# Skills (macOS)
The macOS app surfaces OpenClaw skills via the gateway; it does not parse skills locally.
## Data source
- `skills.status` (gateway) returns all skills plus eligibility and missing requirements
(including allowlist blocks for bundled skills).
- Requirements are derived from `metadata.openclaw.requires` in each `SKILL.md`.
## Install actions
- `metadata.openclaw.install` defines install options (brew/node/go/uv).
- The app calls `skills.install` to run installers on the gateway host.
- The gateway surfaces only one preferred installer when multiple are provided
(brew when available, otherwise node manager from `skills.install`, default npm).
## Env/API keys
- The app stores keys in `~/.openclaw/openclaw.json` under `skills.entries.<skillKey>`.
- `skills.update` patches `enabled`, `apiKey`, and `env`.
## Remote mode
- Install + config updates happen on the gateway host (not the local Mac).

View File

@@ -3,20 +3,24 @@ summary: "Voice overlay lifecycle when wake-word and push-to-talk overlap"
read_when:
- Adjusting voice overlay behavior
---
# Voice Overlay Lifecycle (macOS)
Audience: macOS app contributors. Goal: keep the voice overlay predictable when wake-word and push-to-talk overlap.
### Current intent
- If the overlay is already visible from wake-word and the user presses the hotkey, the hotkey session *adopts* the existing text instead of resetting it. The overlay stays up while the hotkey is held. When the user releases: send if there is trimmed text, otherwise dismiss.
- If the overlay is already visible from wake-word and the user presses the hotkey, the hotkey session _adopts_ the existing text instead of resetting it. The overlay stays up while the hotkey is held. When the user releases: send if there is trimmed text, otherwise dismiss.
- Wake-word alone still auto-sends on silence; push-to-talk sends immediately on release.
### Implemented (Dec 9, 2025)
- Overlay sessions now carry a token per capture (wake-word or push-to-talk). Partial/final/send/dismiss/level updates are dropped when the token doesnt match, avoiding stale callbacks.
- Push-to-talk adopts any visible overlay text as a prefix (so pressing the hotkey while the wake overlay is up keeps the text and appends new speech). It waits up to 1.5s for a final transcript before falling back to the current text.
- Chime/overlay logging is emitted at `info` in categories `voicewake.overlay`, `voicewake.ptt`, and `voicewake.chime` (session start, partial, final, send, dismiss, chime reason).
### Next steps
1. **VoiceSessionCoordinator (actor)**
- Owns exactly one `VoiceSession` at a time.
- API (token-based): `beginWakeCapture`, `beginPushToTalk`, `updatePartial`, `endCapture`, `cancel`, `applyCooldown`.
@@ -36,15 +40,18 @@ Audience: macOS app contributors. Goal: keep the voice overlay predictable when
- Key events: `session_started`, `adopted_by_push_to_talk`, `partial`, `finalized`, `send`, `dismiss`, `cancel`, `cooldown`.
### Debugging checklist
- Stream logs while reproducing a sticky overlay:
```bash
sudo log stream --predicate 'subsystem == "bot.molt" AND category CONTAINS "voicewake"' --level info --style compact
```
- Verify only one active session token; stale callbacks should be dropped by the coordinator.
- Ensure push-to-talk release always calls `endCapture` with the active token; if text is empty, expect `dismiss` without chime or send.
### Migration steps (suggested)
1. Add `VoiceSessionCoordinator`, `VoiceSession`, and `VoiceSessionPublisher`.
2. Refactor `VoiceWakeRuntime` to create/update/end sessions instead of touching `VoiceWakeOverlayController` directly.
3. Refactor `VoicePushToTalk` to adopt existing sessions and call `endCapture` on release; apply runtime cooldown.

View File

@@ -3,14 +3,16 @@ summary: "Voice wake and push-to-talk modes plus routing details in the mac app"
read_when:
- Working on voice wake or PTT pathways
---
# Voice Wake & Push-to-Talk
## Modes
- **Wake-word mode** (default): always-on Speech recognizer waits for trigger tokens (`swabbleTriggerWords`). On match it starts capture, shows the overlay with partial text, and auto-sends after silence.
- **Push-to-talk (Right Option hold)**: hold the right Option key to capture immediately—no trigger needed. The overlay appears while held; releasing finalizes and forwards after a short delay so you can tweak text.
## Runtime behavior (wake-word)
- Speech recognizer lives in `VoiceWakeRuntime`.
- Trigger only fires when theres a **meaningful pause** between the wake word and the next word (~0.55s gap). The overlay/chime can start on the pause even before the command begins.
- Silence windows: 2.0s when speech is flowing, 5.0s if only the trigger was heard.
@@ -20,17 +22,21 @@ read_when:
- After send, recognizer restarts cleanly to listen for the next trigger.
## Lifecycle invariants
- If Voice Wake is enabled and permissions are granted, the wake-word recognizer should be listening (except during an explicit push-to-talk capture).
- Overlay visibility (including manual dismiss via the X button) must never prevent the recognizer from resuming.
## Sticky overlay failure mode (previous)
Previously, if the overlay got stuck visible and you manually closed it, Voice Wake could appear “dead” because the runtimes restart attempt could be blocked by overlay visibility and no subsequent restart was scheduled.
Hardening:
- Wake runtime restart is no longer blocked by overlay visibility.
- Overlay dismiss completion triggers a `VoiceWakeRuntime.refresh(...)` via `VoiceSessionCoordinator`, so manual X-dismiss always resumes listening.
## Push-to-talk specifics
- Hotkey detection uses a global `.flagsChanged` monitor for **right Option** (`keyCode 61` + `.option`). We only observe events (no swallowing).
- Capture pipeline lives in `VoicePushToTalk`: starts Speech immediately, streams partials to the overlay, and calls `VoiceWakeForwarder` on release.
- When push-to-talk starts we pause the wake-word runtime to avoid dueling audio taps; it restarts automatically after release.
@@ -38,6 +44,7 @@ Hardening:
- External keyboards: some may not expose right Option as expected—offer a fallback shortcut if users report misses.
## User-facing settings
- **Voice Wake** toggle: enables wake-word runtime.
- **Hold Cmd+Fn to talk**: enables the push-to-talk monitor. Disabled on macOS < 26.
- Language & mic pickers, live level meter, trigger-word table, tester (local-only; does not forward).
@@ -45,12 +52,15 @@ Hardening:
- **Sounds**: chimes on trigger detect and on send; defaults to the macOS “Glass” system sound. You can pick any `NSSound`-loadable file (e.g. MP3/WAV/AIFF) for each event or choose **No Sound**.
## Forwarding behavior
- When Voice Wake is enabled, transcripts are forwarded to the active gateway/agent (the same local vs remote mode used by the rest of the mac app).
- Replies are delivered to the **last-used main provider** (WhatsApp/Telegram/Discord/WebChat). If delivery fails, the error is logged and the run is still visible via WebChat/session logs.
## Forwarding payload
- `VoiceWakeForwarder.prefixedTranscript(_:)` prepends the machine hint before sending. Shared between wake-word and push-to-talk paths.
## Quick verification
- Toggle push-to-talk on, hold Cmd+Fn, speak, release: overlay should show partials then send.
- While holding, menu-bar ears should stay enlarged (uses `triggerVoiceEars(ttl:nil)`); they drop after release.

View File

@@ -3,6 +3,7 @@ summary: "How the mac app embeds the gateway WebChat and how to debug it"
read_when:
- Debugging mac WebChat view or loopback port
---
# WebChat (macOS app)
The macOS menu bar app embeds the WebChat UI as a native SwiftUI view. It

View File

@@ -3,26 +3,32 @@ summary: "macOS IPC architecture for OpenClaw app, gateway node transport, and P
read_when:
- Editing IPC contracts or menu bar app IPC
---
# OpenClaw macOS IPC architecture
**Current model:** a local Unix socket connects the **node host service** to the **macOS app** for exec approvals + `system.run`. A `openclaw-mac` debug CLI exists for discovery/connect checks; agent actions still flow through the Gateway WebSocket and `node.invoke`. UI automation uses PeekabooBridge.
## Goals
- Single GUI app instance that owns all TCC-facing work (notifications, screen recording, mic, speech, AppleScript).
- A small surface for automation: Gateway + node commands, plus PeekabooBridge for UI automation.
- Predictable permissions: always the same signed bundle ID, launched by launchd, so TCC grants stick.
## How it works
### Gateway + node transport
- The app runs the Gateway (local mode) and connects to it as a node.
- Agent actions are performed via `node.invoke` (e.g. `system.run`, `system.notify`, `canvas.*`).
### Node service + app IPC
- A headless node host service connects to the Gateway WebSocket.
- `system.run` requests are forwarded to the macOS app over a local Unix socket.
- The app performs the exec in UI context, prompts if needed, and returns output.
Diagram (SCI):
```
Agent -> Gateway -> Node Service (WS)
| IPC (UDS + token + HMAC + TTL)
@@ -31,12 +37,14 @@ Agent -> Gateway -> Node Service (WS)
```
### PeekabooBridge (UI automation)
- UI automation uses a separate UNIX socket named `bridge.sock` and the PeekabooBridge JSON protocol.
- Host preference order (client-side): Peekaboo.app → Claude.app → OpenClaw.app → local execution.
- Security: bridge hosts require an allowed TeamID; DEBUG-only same-UID escape hatch is guarded by `PEEKABOO_ALLOW_UNSIGNED_SOCKET_CLIENTS=1` (Peekaboo convention).
- See: [PeekabooBridge usage](/platforms/mac/peekaboo) for details.
## Operational flows
- Restart/rebuild: `SIGN_IDENTITY="Apple Development: <Developer Name> (<TEAMID>)" scripts/restart-mac.sh`
- Kills existing instances
- Swift build + package
@@ -44,6 +52,7 @@ Agent -> Gateway -> Node Service (WS)
- Single instance: app exits early if another instance with the same bundle ID is running.
## Hardening notes
- Prefer requiring a TeamID match for all privileged surfaces.
- PeekabooBridge: `PEEKABOO_ALLOW_UNSIGNED_SOCKET_CLIENTS=1` (DEBUG-only) may allow same-UID callers for local development.
- All communication remains local-only; no network sockets are exposed.