diff --git a/.agents/skills/merge-pr/SKILL.md b/.agents/skills/merge-pr/SKILL.md index ae89b1a2742..041e79a6768 100644 --- a/.agents/skills/merge-pr/SKILL.md +++ b/.agents/skills/merge-pr/SKILL.md @@ -19,6 +19,7 @@ Merge a prepared PR only after deterministic validation. - Never use `gh pr merge --auto` in this flow. - Never run `git push` directly. - Require `--match-head-commit` during merge. +- Wrapper commands are cwd-agnostic; you can run them from repo root or inside the PR worktree. ## Execution Contract diff --git a/.agents/skills/review-pr/SKILL.md b/.agents/skills/review-pr/SKILL.md index ab9d75d967f..f5694ca2c41 100644 --- a/.agents/skills/review-pr/SKILL.md +++ b/.agents/skills/review-pr/SKILL.md @@ -18,6 +18,7 @@ Perform a read-only review and produce both human and machine-readable outputs. - Never push, merge, or modify code intended to keep. - Work only in `.worktrees/pr-`. +- Wrapper commands are cwd-agnostic; you can run them from repo root or inside the PR worktree. ## Execution Contract diff --git a/CHANGELOG.md b/CHANGELOG.md index 2cd3e6abe38..7257b0d2339 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,19 +6,34 @@ Docs: https://docs.openclaw.ai ### Changes +- Agents: add synthetic catalog support for `hf:zai-org/GLM-5`. (#15867) Thanks @battman21. - Skills: remove duplicate `local-places` Google Places skill/proxy and keep `goplaces` as the single supported Google Places path. - Discord: send voice messages with waveform previews from local audio files (including silent delivery). (#7253) Thanks @nyanjou. - Discord: add configurable presence status/activity/type/url (custom status defaults to activity text). (#10855) Thanks @h0tp-ftw. +- Slack/Plugins: add thread-ownership outbound gating via `message_sending` hooks, including @-mention bypass tracking and Slack outbound hook wiring for cancel/modify behavior. (#15775) Thanks @DarlingtonDeveloper. +- Agents: add pre-prompt context diagnostics (`messages`, `systemPromptChars`, `promptChars`, provider/model, session file) before embedded runner prompt calls to improve overflow debugging. (#8930) Thanks @Glucksberg. ### Fixes +- Agents/Compaction: centralize exec default resolution in the shared tool factory so per-agent `tools.exec` overrides (host/security/ask/node and related defaults) persist across compaction retries. (#15833) Thanks @napetrov. +- CLI/Completion: route plugin-load logs to stderr and write generated completion scripts directly to stdout to avoid `source <(openclaw completion ...)` corruption. (#15481) Thanks @arosstale. +- Gateway/Agents: stop injecting a phantom `main` agent into gateway agent listings when `agents.list` explicitly excludes it. (#11450) Thanks @arosstale. +- Agents/Heartbeat: stop auto-creating `HEARTBEAT.md` during workspace bootstrap so missing files continue to run heartbeat as documented. (#11766) Thanks @shadril238. +- Telegram: cap bot menu registration to Telegram's 100-command limit with an overflow warning while keeping typed hidden commands available. (#15844) Thanks @battman21. +- CLI: lazily load outbound provider dependencies and remove forced success-path exits so commands terminate naturally without killing intentional long-running foreground actions. (#12906) Thanks @DrCrinkle. +- Auto-reply/Heartbeat: strip sentence-ending `HEARTBEAT_OK` tokens even when followed by up to 4 punctuation characters, while preserving surrounding sentence punctuation. (#15847) Thanks @Spacefish. +- Heartbeat: allow explicit wake (`wake`) and hook wake (`hook:*`) reasons to run even when `HEARTBEAT.md` is effectively empty so queued system events are processed. (#14527) Thanks @arosstale. - Clawdock: avoid Zsh readonly variable collisions in helper scripts. (#15501) Thanks @nkelner. - Discord: route autoThread replies to existing threads instead of the root channel. (#8302) Thanks @gavinbmoore, @thewilloftheshadow. +- Discord/Agents: apply channel/group `historyLimit` during embedded-runner history compaction to prevent long-running channel sessions from bypassing truncation and overflowing context windows. (#11224) Thanks @shadril238. +- Telegram: scope skill commands to the resolved agent for default accounts so `setMyCommands` no longer triggers `BOT_COMMANDS_TOO_MUCH` when multiple agents are configured. (#15599) +- Plugins/Hooks: fire `before_tool_call` hook exactly once per tool invocation in embedded runs by removing duplicate dispatch paths while preserving parameter mutation semantics. (#15635) Thanks @lailoo. - Agents/Image tool: cap image-analysis completion `maxTokens` by model capability (`min(4096, model.maxTokens)`) to avoid over-limit provider failures while still preventing truncation. (#11770) Thanks @detecti1. - TUI/Streaming: preserve richer streamed assistant text when final payload drops pre-tool-call text blocks, while keeping non-empty final payload authoritative for plain-text updates. (#15452) Thanks @TsekaLuk. - Inbound/Web UI: preserve literal `\n` sequences when normalizing inbound text so Windows paths like `C:\\Work\\nxxx\\README.md` are not corrupted. (#11547) Thanks @mcaxtr. - Daemon/Windows: preserve literal backslashes in `gateway.cmd` command parsing so drive and UNC paths are not corrupted in runtime checks and doctor entrypoint comparisons. (#15642) Thanks @arosstale. - Security/Canvas: serve A2UI assets via the shared safe-open path (`openFileWithinRoot`) to close traversal/TOCTOU gaps, with traversal and symlink regression coverage. (#10525) Thanks @abdelsfane. +- Media: classify `text/*` MIME types as documents in media-kind routing so text attachments are no longer treated as unknown. (#12237) Thanks @arosstale. - Security/Gateway: breaking default-behavior change - canvas IP-based auth fallback now only accepts machine-scoped addresses (RFC1918, link-local, ULA IPv6, CGNAT); public-source IP matches now require bearer token auth. (#14661) Thanks @sumleo. - Security/Gateway: sanitize and truncate untrusted WebSocket header values in pre-handshake close logs to reduce log-poisoning risk. Thanks @thewilloftheshadow. - Security/WhatsApp: enforce `0o600` on `creds.json` and `creds.json.bak` on save/backup/restore paths to reduce credential file exposure. (#10529) Thanks @abdelsfane. @@ -29,13 +44,18 @@ Docs: https://docs.openclaw.ai - MS Teams: preserve parsed mention entities/text when appending OneDrive fallback file links, and accept broader real-world Teams mention ID formats (`29:...`, `8:orgid:...`) while still rejecting placeholder patterns. (#15436) Thanks @hyojin. - Security/Audit: distinguish external webhooks (`hooks.enabled`) from internal hooks (`hooks.internal.enabled`) in attack-surface summaries to avoid false exposure signals when only internal hooks are enabled. (#13474) Thanks @mcaxtr. - Security/Onboarding: clarify multi-user DM isolation remediation with explicit `openclaw config set session.dmScope ...` commands in security audit, doctor security, and channel onboarding guidance. (#13129) Thanks @VintLin. +- Gateway/Hooks: preserve `408` for hook request-body timeout responses while keeping bounded auth-failure cache eviction behavior, with timeout-status regression coverage. (#15848) Thanks @AI-Reviewer-QS. - Security/Audit: add misconfiguration checks for sandbox Docker config with sandbox mode off, ineffective `gateway.nodes.denyCommands` entries, global minimal tool-profile overrides by agent profiles, and permissive extension-plugin tool reachability. - Security/Link understanding: block loopback/internal host patterns and private/mapped IPv6 addresses in extracted URL handling to close SSRF bypasses in link CLI flows. (#15604) Thanks @AI-Reviewer-QS. - Android/Nodes: harden `app.update` by requiring HTTPS and gateway-host URL matching plus SHA-256 verification, stream URL camera downloads to disk with size guards to avoid memory spikes, and stop signing release builds with debug keys. (#13541) Thanks @smartprogrammer93. - Auto-reply/Threading: auto-inject implicit reply threading so `replyToMode` works without requiring model-emitted `[[reply_to_current]]`, while preserving `replyToMode: "off"` behavior for implicit Slack replies and keeping block-streaming chunk coalescing stable under `replyToMode: "first"`. (#14976) Thanks @Diaspar4u. +- Auto-reply/Media: allow image-only inbound messages (no caption) to reach the agent instead of short-circuiting as empty text, and preserve thread context in queued/followup prompt bodies for media-only runs. (#11916) Thanks @arosstale. - Sandbox: pass configured `sandbox.docker.env` variables to sandbox containers at `docker create` time. (#15138) Thanks @stevebot-alive. +- Gateway/Restart: clear stale command-queue and heartbeat wake runtime state after SIGUSR1 in-process restarts to prevent zombie gateway behavior where queued work stops draining. (#15195) Thanks @joeykrug. - Onboarding/CLI: restore terminal state without resuming paused `stdin`, so onboarding exits cleanly after choosing Web UI and the installer returns instead of appearing stuck. - Auth/OpenAI Codex: share OAuth login handling across onboarding and `models auth login --provider openai-codex`, keep onboarding alive when OAuth fails, and surface a direct OAuth help note instead of terminating the wizard. (#15406, follow-up to #14552) Thanks @zhiluo20. +- Cron: add regression coverage for announce-mode isolated jobs so runs that already report `delivered: true` do not enqueue duplicate main-session relays, including delivery configs where `mode` is omitted and defaults to announce. (#15737) Thanks @brandonwise. +- Cron: honor `deleteAfterRun` in isolated announce delivery by mapping it to subagent announce cleanup mode, so cron run sessions configured for deletion are removed after completion. (#15368) Thanks @arosstale. - Onboarding/Providers: add vLLM as an onboarding provider with model discovery, auth profile wiring, and non-interactive auth-choice validation. (#12577) Thanks @gejifeng. - Onboarding/Providers: preserve Hugging Face auth intent in auth-choice remapping (`tokenProvider=huggingface` with `authChoice=apiKey`) and skip env-override prompts when an explicit token is provided. (#13472) Thanks @Josephrp. - OpenAI Codex/Spark: implement end-to-end `gpt-5.3-codex-spark` support across fallback/thinking/model resolution and `models list` forward-compat visibility. (#14990, #15174) Thanks @L-U-C-K-Y, @loiie45e. @@ -47,6 +67,7 @@ Docs: https://docs.openclaw.ai - macOS Voice Wake: fix a crash in trigger trimming for CJK/Unicode transcripts by matching and slicing on original-string ranges instead of transformed-string indices. (#11052) Thanks @Flash-LHR. - Heartbeat: prevent scheduler silent-death races during runner reloads, preserve retry cooldown backoff under wake bursts, and prioritize user/action wake causes over interval/retry reasons when coalescing. (#15108) Thanks @joeykrug. - Outbound targets: fail closed for WhatsApp/Twitch/Google Chat fallback paths so invalid or missing targets are dropped instead of rerouted, and align resolver hints with strict target requirements. (#13578) Thanks @mcaxtr. +- Outbound: add a write-ahead delivery queue with crash-recovery retries to prevent lost outbound messages after gateway restarts. (#15636) Thanks @nabbilkhan, @thewilloftheshadow. - Exec/Allowlist: allow multiline heredoc bodies (`<<`, `<<-`) while keeping multiline non-heredoc shell commands blocked, so exec approval parsing permits heredoc input safely without allowing general newline command chaining. (#13811) Thanks @mcaxtr. - Docs/Mermaid: remove hardcoded Mermaid init theme blocks from four docs diagrams so dark mode inherits readable theme defaults. (#15157) Thanks @heytulsiprasad. - Outbound/Threading: pass `replyTo` and `threadId` from `message send` tool actions through the core outbound send path to channel adapters, preserving thread/reply routing. (#14948) Thanks @mcaxtr. @@ -56,12 +77,16 @@ Docs: https://docs.openclaw.ai - Signal/Install: auto-install `signal-cli` via Homebrew on non-x64 Linux architectures, avoiding x86_64 native binary `Exec format error` failures on arm64/arm hosts. (#15443) Thanks @jogvan-k. - Discord: avoid misrouting numeric guild allowlist entries to `/channels/` by prefixing guild-only inputs with `guild:` during resolution. (#12326) Thanks @headswim. - Config: preserve `${VAR}` env references when writing config files so `openclaw config set/apply/patch` does not persist secrets to disk. Thanks @thewilloftheshadow. +- Config: remove a cross-request env-snapshot race in config writes by carrying read-time env context into write calls per request, preserving `${VAR}` refs safely under concurrent gateway config mutations. (#11560) Thanks @akoscz. +- Config: log overwrite audit entries (path, backup target, and hash transition) whenever an existing config file is replaced, improving traceability for unexpected config clobbers. - Process/Exec: avoid shell execution for `.exe` commands on Windows so env overrides work reliably in `runCommandWithTimeout`. Thanks @thewilloftheshadow. - Web tools/web_fetch: prefer `text/markdown` responses for Cloudflare Markdown for Agents, add `cf-markdown` extraction for markdown bodies, and redact fetched URLs in `x-markdown-tokens` debug logs to avoid leaking raw paths/query params. (#15376) Thanks @Yaxuan42. - Config: keep legacy audio transcription migration strict by rejecting non-string/unsafe command tokens while still migrating valid custom script executables. (#5042) Thanks @shayan919293. - Status/Sessions: stop clamping derived `totalTokens` to context-window size, keep prompt-token snapshots wired through session accounting, and surface context usage as unknown when fresh snapshot data is missing to avoid false 100% reports. (#15114) Thanks @echoVic. - Providers/MiniMax: switch implicit MiniMax API-key provider from `openai-completions` to `anthropic-messages` with the correct Anthropic-compatible base URL, fixing `invalid role: developer (2013)` errors on MiniMax M2.5. (#15275) Thanks @lailoo. - Config: accept `$schema` key in config file so JSON Schema editor tooling works without validation errors. (#14998) +- Web UI: add `img` to DOMPurify allowed tags and `src`/`alt` to allowed attributes so markdown images render in webchat instead of being stripped. (#15437) Thanks @lailoo. +- Ollama/Agents: use resolved model/provider base URLs for native `/api/chat` streaming (including aliased providers), normalize `/v1` endpoints, and forward abort + `maxTokens` stream options for reliable cancellation and token caps. (#11853) Thanks @BrokenFinger98. ## 2026.2.12 @@ -103,6 +128,7 @@ Docs: https://docs.openclaw.ai - Cron: pass `agentId` to `runHeartbeatOnce` for main-session jobs. (#14140) Thanks @ishikawa-pro. - Cron: re-arm timers when `onTimer` fires while a job is still executing. (#14233) Thanks @tomron87. - Cron: prevent duplicate fires when multiple jobs trigger simultaneously. (#14256) Thanks @xinhuagu. +- Cron: prevent duplicate announce-mode isolated cron deliveries, and keep main-session fallback active when best-effort structured delivery attempts fail to send any message. (#15739) Thanks @widingmarcus-cyber. - Cron: isolate scheduler errors so one bad job does not break all jobs. (#14385) Thanks @MarvinDontPanic. - Cron: prevent one-shot `at` jobs from re-firing on restart after skipped/errored runs. (#13878) Thanks @lailoo. - Heartbeat: prevent scheduler stalls on unexpected run errors and avoid immediate rerun loops after `requests-in-flight` skips. (#14901) Thanks @joeykrug. @@ -131,6 +157,7 @@ Docs: https://docs.openclaw.ai - Onboarding/Providers: update MiniMax API default/recommended models from M2.1 to M2.5, add M2.5/M2.5-Lightning model entries, and include `minimax-m2.5` in modern model filtering. (#14865) Thanks @adao-max. - Ollama: use configured `models.providers.ollama.baseUrl` for model discovery and normalize `/v1` endpoints to the native Ollama API root. (#14131) Thanks @shtse8. - Voice Call: pass Twilio stream auth token via `` instead of query string. (#14029) Thanks @mcwigglesmcgee. +- Config/Models: allow full `models.providers.*.models[*].compat` keys used by `openai-completions` (`thinkingFormat`, `supportsStrictMode`, and streaming/tool-result compatibility flags) so valid provider overrides no longer fail strict config validation. (#11063) Thanks @ikari-pl. - Feishu: pass `Buffer` directly to the Feishu SDK upload APIs instead of `Readable.from(...)` to avoid form-data upload failures. (#10345) Thanks @youngerstyle. - Feishu: trigger mention-gated group handling only when the bot itself is mentioned (not just any mention). (#11088) Thanks @openperf. - Feishu: probe status uses the resolved account context for multi-account credential checks. (#11233) Thanks @onevcat. @@ -156,6 +183,7 @@ Docs: https://docs.openclaw.ai - Agents: keep followup-runner session `totalTokens` aligned with post-compaction context by using last-call usage and shared token-accounting logic. (#14979) Thanks @shtse8. - Hooks/Plugins: wire 9 previously unwired plugin lifecycle hooks into core runtime paths (session, compaction, gateway, and outbound message hooks). (#14882) Thanks @shtse8. - Hooks/Tools: dispatch `before_tool_call` and `after_tool_call` hooks from both tool execution paths with rebased conflict fixes. (#15012) Thanks @Patrick-Barletta, @Takhoffman. +- Hooks: replace loader `console.*` output with subsystem logger messages so hook loading errors/warnings route through standard logging. (#11029) Thanks @shadril238. - Discord: allow channel-edit to archive/lock threads and set auto-archive duration. (#5542) Thanks @stumct. - Discord tests: use a partial @buape/carbon mock in slash command coverage. (#13262) Thanks @arosstale. - Tests: update thread ID handling in Slack message collection tests. (#14108) Thanks @swizzmagik. diff --git a/apps/macos/Sources/OpenClaw/GatewayEndpointStore.swift b/apps/macos/Sources/OpenClaw/GatewayEndpointStore.swift index 20961e379bf..0edb2e65122 100644 --- a/apps/macos/Sources/OpenClaw/GatewayEndpointStore.swift +++ b/apps/macos/Sources/OpenClaw/GatewayEndpointStore.swift @@ -619,7 +619,29 @@ actor GatewayEndpointStore { } extension GatewayEndpointStore { - static func dashboardURL(for config: GatewayConnection.Config) throws -> URL { + private static func normalizeDashboardPath(_ rawPath: String?) -> String { + let trimmed = (rawPath ?? "").trimmingCharacters(in: .whitespacesAndNewlines) + guard !trimmed.isEmpty else { return "/" } + let withLeadingSlash = trimmed.hasPrefix("/") ? trimmed : "/" + trimmed + guard withLeadingSlash != "/" else { return "/" } + return withLeadingSlash.hasSuffix("/") ? withLeadingSlash : withLeadingSlash + "/" + } + + private static func localControlUiBasePath() -> String { + let root = OpenClawConfigFile.loadDict() + guard let gateway = root["gateway"] as? [String: Any], + let controlUi = gateway["controlUi"] as? [String: Any] + else { + return "/" + } + return self.normalizeDashboardPath(controlUi["basePath"] as? String) + } + + static func dashboardURL( + for config: GatewayConnection.Config, + mode: AppState.ConnectionMode, + localBasePath: String? = nil) throws -> URL + { guard var components = URLComponents(url: config.url, resolvingAgainstBaseURL: false) else { throw NSError(domain: "Dashboard", code: 1, userInfo: [ NSLocalizedDescriptionKey: "Invalid gateway URL", @@ -633,7 +655,17 @@ extension GatewayEndpointStore { default: components.scheme = "http" } - components.path = "/" + + let urlPath = self.normalizeDashboardPath(components.path) + if urlPath != "/" { + components.path = urlPath + } else if mode == .local { + let fallbackPath = localBasePath ?? self.localControlUiBasePath() + components.path = self.normalizeDashboardPath(fallbackPath) + } else { + components.path = "/" + } + var queryItems: [URLQueryItem] = [] if let token = config.token?.trimmingCharacters(in: .whitespacesAndNewlines), !token.isEmpty diff --git a/apps/macos/Sources/OpenClaw/MenuContentView.swift b/apps/macos/Sources/OpenClaw/MenuContentView.swift index 6dec4d93620..fd1b437cf7c 100644 --- a/apps/macos/Sources/OpenClaw/MenuContentView.swift +++ b/apps/macos/Sources/OpenClaw/MenuContentView.swift @@ -337,7 +337,7 @@ struct MenuContent: View { private func openDashboard() async { do { let config = try await GatewayEndpointStore.shared.requireConfig() - let url = try GatewayEndpointStore.dashboardURL(for: config) + let url = try GatewayEndpointStore.dashboardURL(for: config, mode: self.state.connectionMode) NSWorkspace.shared.open(url) } catch { let alert = NSAlert() diff --git a/apps/macos/Sources/OpenClaw/OpenClawConfigFile.swift b/apps/macos/Sources/OpenClaw/OpenClawConfigFile.swift index 3f7d3c03aa5..fc66030e3f5 100644 --- a/apps/macos/Sources/OpenClaw/OpenClawConfigFile.swift +++ b/apps/macos/Sources/OpenClaw/OpenClawConfigFile.swift @@ -3,6 +3,7 @@ import Foundation enum OpenClawConfigFile { private static let logger = Logger(subsystem: "ai.openclaw", category: "config") + private static let configAuditFileName = "config-audit.jsonl" static func url() -> URL { OpenClawPaths.configURL @@ -35,15 +36,61 @@ enum OpenClawConfigFile { static func saveDict(_ dict: [String: Any]) { // Nix mode disables config writes in production, but tests rely on saving temp configs. if ProcessInfo.processInfo.isNixMode, !ProcessInfo.processInfo.isRunningTests { return } + let url = self.url() + let previousData = try? Data(contentsOf: url) + let previousRoot = previousData.flatMap { self.parseConfigData($0) } + let previousBytes = previousData?.count + let hadMetaBefore = self.hasMeta(previousRoot) + let gatewayModeBefore = self.gatewayMode(previousRoot) + + var output = dict + self.stampMeta(&output) + do { - let data = try JSONSerialization.data(withJSONObject: dict, options: [.prettyPrinted, .sortedKeys]) - let url = self.url() + let data = try JSONSerialization.data(withJSONObject: output, options: [.prettyPrinted, .sortedKeys]) try FileManager().createDirectory( at: url.deletingLastPathComponent(), withIntermediateDirectories: true) try data.write(to: url, options: [.atomic]) + let nextBytes = data.count + let gatewayModeAfter = self.gatewayMode(output) + let suspicious = self.configWriteSuspiciousReasons( + existsBefore: previousData != nil, + previousBytes: previousBytes, + nextBytes: nextBytes, + hadMetaBefore: hadMetaBefore, + gatewayModeBefore: gatewayModeBefore, + gatewayModeAfter: gatewayModeAfter) + if !suspicious.isEmpty { + self.logger.warning("config write anomaly (\(suspicious.joined(separator: ", "))) at \(url.path)") + } + self.appendConfigWriteAudit([ + "result": "success", + "configPath": url.path, + "existsBefore": previousData != nil, + "previousBytes": previousBytes ?? NSNull(), + "nextBytes": nextBytes, + "hasMetaBefore": hadMetaBefore, + "hasMetaAfter": self.hasMeta(output), + "gatewayModeBefore": gatewayModeBefore ?? NSNull(), + "gatewayModeAfter": gatewayModeAfter ?? NSNull(), + "suspicious": suspicious, + ]) } catch { self.logger.error("config save failed: \(error.localizedDescription)") + self.appendConfigWriteAudit([ + "result": "failed", + "configPath": url.path, + "existsBefore": previousData != nil, + "previousBytes": previousBytes ?? NSNull(), + "nextBytes": NSNull(), + "hasMetaBefore": hadMetaBefore, + "hasMetaAfter": self.hasMeta(output), + "gatewayModeBefore": gatewayModeBefore ?? NSNull(), + "gatewayModeAfter": self.gatewayMode(output) ?? NSNull(), + "suspicious": [], + "error": error.localizedDescription, + ]) } } @@ -214,4 +261,100 @@ enum OpenClawConfigFile { } return nil } + + private static func stampMeta(_ root: inout [String: Any]) { + var meta = root["meta"] as? [String: Any] ?? [:] + let version = Bundle.main.object(forInfoDictionaryKey: "CFBundleShortVersionString") as? String ?? "macos-app" + meta["lastTouchedVersion"] = version + meta["lastTouchedAt"] = ISO8601DateFormatter().string(from: Date()) + root["meta"] = meta + } + + private static func hasMeta(_ root: [String: Any]?) -> Bool { + guard let root else { return false } + return root["meta"] is [String: Any] + } + + private static func hasMeta(_ root: [String: Any]) -> Bool { + root["meta"] is [String: Any] + } + + private static func gatewayMode(_ root: [String: Any]?) -> String? { + guard let root else { return nil } + return self.gatewayMode(root) + } + + private static func gatewayMode(_ root: [String: Any]) -> String? { + guard let gateway = root["gateway"] as? [String: Any], + let mode = gateway["mode"] as? String + else { return nil } + let trimmed = mode.trimmingCharacters(in: .whitespacesAndNewlines) + return trimmed.isEmpty ? nil : trimmed + } + + private static func configWriteSuspiciousReasons( + existsBefore: Bool, + previousBytes: Int?, + nextBytes: Int, + hadMetaBefore: Bool, + gatewayModeBefore: String?, + gatewayModeAfter: String?) -> [String] + { + var reasons: [String] = [] + if !existsBefore { + return reasons + } + if let previousBytes, previousBytes >= 512, nextBytes < max(1, previousBytes / 2) { + reasons.append("size-drop:\(previousBytes)->\(nextBytes)") + } + if !hadMetaBefore { + reasons.append("missing-meta-before-write") + } + if gatewayModeBefore != nil, gatewayModeAfter == nil { + reasons.append("gateway-mode-removed") + } + return reasons + } + + private static func configAuditLogURL() -> URL { + self.stateDirURL() + .appendingPathComponent("logs", isDirectory: true) + .appendingPathComponent(self.configAuditFileName, isDirectory: false) + } + + private static func appendConfigWriteAudit(_ fields: [String: Any]) { + var record: [String: Any] = [ + "ts": ISO8601DateFormatter().string(from: Date()), + "source": "macos-openclaw-config-file", + "event": "config.write", + "pid": ProcessInfo.processInfo.processIdentifier, + "argv": Array(ProcessInfo.processInfo.arguments.prefix(8)), + ] + for (key, value) in fields { + record[key] = value is NSNull ? NSNull() : value + } + guard JSONSerialization.isValidJSONObject(record), + let data = try? JSONSerialization.data(withJSONObject: record) + else { + return + } + var line = Data() + line.append(data) + line.append(0x0A) + let logURL = self.configAuditLogURL() + do { + try FileManager().createDirectory( + at: logURL.deletingLastPathComponent(), + withIntermediateDirectories: true) + if !FileManager().fileExists(atPath: logURL.path) { + FileManager().createFile(atPath: logURL.path, contents: nil) + } + let handle = try FileHandle(forWritingTo: logURL) + defer { try? handle.close() } + try handle.seekToEnd() + try handle.write(contentsOf: line) + } catch { + // best-effort + } + } } diff --git a/apps/macos/Tests/OpenClawIPCTests/GatewayEndpointStoreTests.swift b/apps/macos/Tests/OpenClawIPCTests/GatewayEndpointStoreTests.swift index 8ab50b6535f..44c464c449f 100644 --- a/apps/macos/Tests/OpenClawIPCTests/GatewayEndpointStoreTests.swift +++ b/apps/macos/Tests/OpenClawIPCTests/GatewayEndpointStoreTests.swift @@ -176,6 +176,48 @@ import Testing #expect(host == "192.168.1.10") } + @Test func dashboardURLUsesLocalBasePathInLocalMode() throws { + let config: GatewayConnection.Config = ( + url: try #require(URL(string: "ws://127.0.0.1:18789")), + token: nil, + password: nil + ) + + let url = try GatewayEndpointStore.dashboardURL( + for: config, + mode: .local, + localBasePath: " control ") + #expect(url.absoluteString == "http://127.0.0.1:18789/control/") + } + + @Test func dashboardURLSkipsLocalBasePathInRemoteMode() throws { + let config: GatewayConnection.Config = ( + url: try #require(URL(string: "ws://gateway.example:18789")), + token: nil, + password: nil + ) + + let url = try GatewayEndpointStore.dashboardURL( + for: config, + mode: .remote, + localBasePath: "/local-ui") + #expect(url.absoluteString == "http://gateway.example:18789/") + } + + @Test func dashboardURLPrefersPathFromConfigURL() throws { + let config: GatewayConnection.Config = ( + url: try #require(URL(string: "wss://gateway.example:443/remote-ui")), + token: nil, + password: nil + ) + + let url = try GatewayEndpointStore.dashboardURL( + for: config, + mode: .remote, + localBasePath: "/local-ui") + #expect(url.absoluteString == "https://gateway.example:443/remote-ui/") + } + @Test func normalizeGatewayUrlAddsDefaultPortForWs() { let url = GatewayRemoteConfig.normalizeGatewayUrl("ws://gateway") #expect(url?.port == 18789) diff --git a/apps/macos/Tests/OpenClawIPCTests/OpenClawConfigFileTests.swift b/apps/macos/Tests/OpenClawIPCTests/OpenClawConfigFileTests.swift index c03505e2f4c..98e4e8046d3 100644 --- a/apps/macos/Tests/OpenClawIPCTests/OpenClawConfigFileTests.swift +++ b/apps/macos/Tests/OpenClawIPCTests/OpenClawConfigFileTests.swift @@ -76,4 +76,43 @@ struct OpenClawConfigFileTests { #expect(OpenClawConfigFile.url().path == "\(dir)/openclaw.json") } } + + @MainActor + @Test + func saveDictAppendsConfigAuditLog() async throws { + let stateDir = FileManager().temporaryDirectory + .appendingPathComponent("openclaw-state-\(UUID().uuidString)", isDirectory: true) + let configPath = stateDir.appendingPathComponent("openclaw.json") + let auditPath = stateDir.appendingPathComponent("logs/config-audit.jsonl") + + defer { try? FileManager().removeItem(at: stateDir) } + + try await TestIsolation.withEnvValues([ + "OPENCLAW_STATE_DIR": stateDir.path, + "OPENCLAW_CONFIG_PATH": configPath.path, + ]) { + OpenClawConfigFile.saveDict([ + "gateway": ["mode": "local"], + ]) + + let configData = try Data(contentsOf: configPath) + let configRoot = try JSONSerialization.jsonObject(with: configData) as? [String: Any] + #expect((configRoot?["meta"] as? [String: Any]) != nil) + + let rawAudit = try String(contentsOf: auditPath, encoding: .utf8) + let lines = rawAudit + .split(whereSeparator: \.isNewline) + .map(String.init) + #expect(!lines.isEmpty) + guard let last = lines.last else { + Issue.record("Missing config audit line") + return + } + let auditRoot = try JSONSerialization.jsonObject(with: Data(last.utf8)) as? [String: Any] + #expect(auditRoot?["source"] as? String == "macos-openclaw-config-file") + #expect(auditRoot?["event"] as? String == "config.write") + #expect(auditRoot?["result"] as? String == "success") + #expect(auditRoot?["configPath"] as? String == configPath.path) + } + } } diff --git a/docs/automation/hooks.md b/docs/automation/hooks.md index 2030e9aeaf6..68c583a7a84 100644 --- a/docs/automation/hooks.md +++ b/docs/automation/hooks.md @@ -41,9 +41,10 @@ The hooks system allows you to: ### Bundled Hooks -OpenClaw ships with three bundled hooks that are automatically discovered: +OpenClaw ships with four bundled hooks that are automatically discovered: - **πŸ’Ύ session-memory**: Saves session context to your agent workspace (default `~/.openclaw/workspace/memory/`) when you issue `/new` +- **πŸ“Ž bootstrap-extra-files**: Injects additional workspace bootstrap files from configured glob/path patterns during `agent:bootstrap` - **πŸ“ command-logger**: Logs all command events to `~/.openclaw/logs/commands.log` - **πŸš€ boot-md**: Runs `BOOT.md` when the gateway starts (requires internal hooks enabled) @@ -484,6 +485,47 @@ Saves session context to memory when you issue `/new`. openclaw hooks enable session-memory ``` +### bootstrap-extra-files + +Injects additional bootstrap files (for example monorepo-local `AGENTS.md` / `TOOLS.md`) during `agent:bootstrap`. + +**Events**: `agent:bootstrap` + +**Requirements**: `workspace.dir` must be configured + +**Output**: No files written; bootstrap context is modified in-memory only. + +**Config**: + +```json +{ + "hooks": { + "internal": { + "enabled": true, + "entries": { + "bootstrap-extra-files": { + "enabled": true, + "paths": ["packages/*/AGENTS.md", "packages/*/TOOLS.md"] + } + } + } + } +} +``` + +**Notes**: + +- Paths are resolved relative to workspace. +- Files must stay inside workspace (realpath-checked). +- Only recognized bootstrap basenames are loaded. +- Subagent allowlist is preserved (`AGENTS.md` and `TOOLS.md` only). + +**Enable**: + +```bash +openclaw hooks enable bootstrap-extra-files +``` + ### command-logger Logs all command events to a centralized audit file. @@ -618,6 +660,7 @@ The gateway logs hook loading at startup: ``` Registered hook: session-memory -> command:new +Registered hook: bootstrap-extra-files -> agent:bootstrap Registered hook: command-logger -> command Registered hook: boot-md -> gateway:startup ``` diff --git a/docs/cli/hooks.md b/docs/cli/hooks.md index 6b4f42143e9..fdf72f83434 100644 --- a/docs/cli/hooks.md +++ b/docs/cli/hooks.md @@ -32,10 +32,11 @@ List all discovered hooks from workspace, managed, and bundled directories. **Example output:** ``` -Hooks (3/3 ready) +Hooks (4/4 ready) Ready: πŸš€ boot-md βœ“ - Run BOOT.md on gateway startup + πŸ“Ž bootstrap-extra-files βœ“ - Inject extra workspace bootstrap files during agent bootstrap πŸ“ command-logger βœ“ - Log all command events to a centralized audit file πŸ’Ύ session-memory βœ“ - Save session context to memory when /new command is issued ``` @@ -249,6 +250,18 @@ openclaw hooks enable session-memory **See:** [session-memory documentation](/automation/hooks#session-memory) +### bootstrap-extra-files + +Injects additional bootstrap files (for example monorepo-local `AGENTS.md` / `TOOLS.md`) during `agent:bootstrap`. + +**Enable:** + +```bash +openclaw hooks enable bootstrap-extra-files +``` + +**See:** [bootstrap-extra-files documentation](/automation/hooks#bootstrap-extra-files) + ### command-logger Logs all command events to a centralized audit file. diff --git a/docs/concepts/compaction.md b/docs/concepts/compaction.md index 54b3d30ecab..cc6effb7e64 100644 --- a/docs/concepts/compaction.md +++ b/docs/concepts/compaction.md @@ -21,7 +21,7 @@ Compaction **persists** in the session’s JSONL history. ## Configuration -See [Compaction config & modes](/concepts/compaction) for the `agents.defaults.compaction` settings. +Use the `agents.defaults.compaction` setting in your `openclaw.json` to configure compaction behavior (mode, target tokens, etc.). ## Auto-compaction (default on) diff --git a/docs/providers/ollama.md b/docs/providers/ollama.md index 463923fb7c2..c6a0e2372e6 100644 --- a/docs/providers/ollama.md +++ b/docs/providers/ollama.md @@ -8,7 +8,7 @@ title: "Ollama" # Ollama -Ollama is a local LLM runtime that makes it easy to run open-source models on your machine. OpenClaw integrates with Ollama's OpenAI-compatible API and can **auto-discover tool-capable models** when you opt in with `OLLAMA_API_KEY` (or an auth profile) and do not define an explicit `models.providers.ollama` entry. +Ollama is a local LLM runtime that makes it easy to run open-source models on your machine. OpenClaw integrates with Ollama's native API (`/api/chat`), supporting streaming and tool calling, and can **auto-discover tool-capable models** when you opt in with `OLLAMA_API_KEY` (or an auth profile) and do not define an explicit `models.providers.ollama` entry. ## Quick start @@ -101,10 +101,9 @@ Use explicit config when: models: { providers: { ollama: { - // Use a host that includes /v1 for OpenAI-compatible APIs - baseUrl: "http://ollama-host:11434/v1", + baseUrl: "http://ollama-host:11434", apiKey: "ollama-local", - api: "openai-completions", + api: "ollama", models: [ { id: "gpt-oss:20b", @@ -134,7 +133,7 @@ If Ollama is running on a different host or port (explicit config disables auto- providers: { ollama: { apiKey: "ollama-local", - baseUrl: "http://ollama-host:11434/v1", + baseUrl: "http://ollama-host:11434", }, }, }, @@ -174,45 +173,28 @@ Ollama is free and runs locally, so all model costs are set to $0. ### Streaming Configuration -Due to a [known issue](https://github.com/badlogic/pi-mono/issues/1205) in the underlying SDK with Ollama's response format, **streaming is disabled by default** for Ollama models. This prevents corrupted responses when using tool-capable models. +OpenClaw's Ollama integration uses the **native Ollama API** (`/api/chat`) by default, which fully supports streaming and tool calling simultaneously. No special configuration is needed. -When streaming is disabled, responses are delivered all at once (non-streaming mode), which avoids the issue where interleaved content/reasoning deltas cause garbled output. +#### Legacy OpenAI-Compatible Mode -#### Re-enable Streaming (Advanced) - -If you want to re-enable streaming for Ollama (may cause issues with tool-capable models): +If you need to use the OpenAI-compatible endpoint instead (e.g., behind a proxy that only supports OpenAI format), set `api: "openai-completions"` explicitly: ```json5 { - agents: { - defaults: { - models: { - "ollama/gpt-oss:20b": { - streaming: true, - }, - }, - }, - }, + models: { + providers: { + ollama: { + baseUrl: "http://ollama-host:11434/v1", + api: "openai-completions", + apiKey: "ollama-local", + models: [...] + } + } + } } ``` -#### Disable Streaming for Other Providers - -You can also disable streaming for any provider if needed: - -```json5 -{ - agents: { - defaults: { - models: { - "openai/gpt-4": { - streaming: false, - }, - }, - }, - }, -} -``` +Note: The OpenAI-compatible endpoint may not support streaming + tool calling simultaneously. You may need to disable streaming with `params: { streaming: false }` in model config. ### Context windows @@ -261,15 +243,6 @@ ps aux | grep ollama ollama serve ``` -### Corrupted responses or tool names in output - -If you see garbled responses containing tool names (like `sessions_send`, `memory_get`) or fragmented text when using Ollama models, this is due to an upstream SDK issue with streaming responses. **This is fixed by default** in the latest OpenClaw version by disabling streaming for Ollama models. - -If you manually enabled streaming and experience this issue: - -1. Remove the `streaming: true` configuration from your Ollama model entries, or -2. Explicitly set `streaming: false` for Ollama models (see [Streaming Configuration](#streaming-configuration)) - ## See Also - [Model Providers](/concepts/model-providers) - Overview of all providers diff --git a/extensions/matrix/src/matrix/send/media.ts b/extensions/matrix/src/matrix/send/media.ts index c4339d90057..eecdce3d565 100644 --- a/extensions/matrix/src/matrix/send/media.ts +++ b/extensions/matrix/src/matrix/send/media.ts @@ -6,7 +6,6 @@ import type { TimedFileInfo, VideoFileInfo, } from "@vector-im/matrix-bot-sdk"; -import { parseBuffer, type IFileInfo } from "music-metadata"; import { getMatrixRuntime } from "../../runtime.js"; import { applyMatrixFormatting } from "./formatting.js"; import { @@ -18,6 +17,7 @@ import { } from "./types.js"; const getCore = () => getMatrixRuntime(); +type IFileInfo = import("music-metadata").IFileInfo; export function buildMatrixMediaInfo(params: { size: number; @@ -164,6 +164,7 @@ export async function resolveMediaDurationMs(params: { return undefined; } try { + const { parseBuffer } = await import("music-metadata"); const fileInfo: IFileInfo | string | undefined = params.contentType || params.fileName ? { diff --git a/extensions/thread-ownership/index.test.ts b/extensions/thread-ownership/index.test.ts new file mode 100644 index 00000000000..3690938a1b0 --- /dev/null +++ b/extensions/thread-ownership/index.test.ts @@ -0,0 +1,180 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import register from "./index.js"; + +describe("thread-ownership plugin", () => { + const hooks: Record = {}; + const api = { + pluginConfig: {}, + config: { + agents: { + list: [{ id: "test-agent", default: true, identity: { name: "TestBot" } }], + }, + }, + id: "thread-ownership", + name: "Thread Ownership", + logger: { info: vi.fn(), warn: vi.fn(), debug: vi.fn() }, + on: vi.fn((hookName: string, handler: Function) => { + hooks[hookName] = handler; + }), + }; + + let originalFetch: typeof globalThis.fetch; + + beforeEach(() => { + vi.clearAllMocks(); + for (const key of Object.keys(hooks)) delete hooks[key]; + + process.env.SLACK_FORWARDER_URL = "http://localhost:8750"; + process.env.SLACK_BOT_USER_ID = "U999"; + + originalFetch = globalThis.fetch; + globalThis.fetch = vi.fn(); + }); + + afterEach(() => { + globalThis.fetch = originalFetch; + delete process.env.SLACK_FORWARDER_URL; + delete process.env.SLACK_BOT_USER_ID; + vi.restoreAllMocks(); + }); + + it("registers message_received and message_sending hooks", () => { + register(api as any); + + expect(api.on).toHaveBeenCalledTimes(2); + expect(api.on).toHaveBeenCalledWith("message_received", expect.any(Function)); + expect(api.on).toHaveBeenCalledWith("message_sending", expect.any(Function)); + }); + + describe("message_sending", () => { + beforeEach(() => { + register(api as any); + }); + + it("allows non-slack channels", async () => { + const result = await hooks.message_sending( + { content: "hello", metadata: { threadTs: "1234.5678", channelId: "C123" }, to: "C123" }, + { channelId: "discord", conversationId: "C123" }, + ); + + expect(result).toBeUndefined(); + expect(globalThis.fetch).not.toHaveBeenCalled(); + }); + + it("allows top-level messages (no threadTs)", async () => { + const result = await hooks.message_sending( + { content: "hello", metadata: {}, to: "C123" }, + { channelId: "slack", conversationId: "C123" }, + ); + + expect(result).toBeUndefined(); + expect(globalThis.fetch).not.toHaveBeenCalled(); + }); + + it("claims ownership successfully", async () => { + vi.mocked(globalThis.fetch).mockResolvedValue( + new Response(JSON.stringify({ owner: "test-agent" }), { status: 200 }), + ); + + const result = await hooks.message_sending( + { content: "hello", metadata: { threadTs: "1234.5678", channelId: "C123" }, to: "C123" }, + { channelId: "slack", conversationId: "C123" }, + ); + + expect(result).toBeUndefined(); + expect(globalThis.fetch).toHaveBeenCalledWith( + "http://localhost:8750/api/v1/ownership/C123/1234.5678", + expect.objectContaining({ + method: "POST", + body: JSON.stringify({ agent_id: "test-agent" }), + }), + ); + }); + + it("cancels when thread owned by another agent", async () => { + vi.mocked(globalThis.fetch).mockResolvedValue( + new Response(JSON.stringify({ owner: "other-agent" }), { status: 409 }), + ); + + const result = await hooks.message_sending( + { content: "hello", metadata: { threadTs: "1234.5678", channelId: "C123" }, to: "C123" }, + { channelId: "slack", conversationId: "C123" }, + ); + + expect(result).toEqual({ cancel: true }); + expect(api.logger.info).toHaveBeenCalledWith(expect.stringContaining("cancelled send")); + }); + + it("fails open on network error", async () => { + vi.mocked(globalThis.fetch).mockRejectedValue(new Error("ECONNREFUSED")); + + const result = await hooks.message_sending( + { content: "hello", metadata: { threadTs: "1234.5678", channelId: "C123" }, to: "C123" }, + { channelId: "slack", conversationId: "C123" }, + ); + + expect(result).toBeUndefined(); + expect(api.logger.warn).toHaveBeenCalledWith( + expect.stringContaining("ownership check failed"), + ); + }); + }); + + describe("message_received @-mention tracking", () => { + beforeEach(() => { + register(api as any); + }); + + it("tracks @-mentions and skips ownership check for mentioned threads", async () => { + // Simulate receiving a message that @-mentions the agent. + await hooks.message_received( + { content: "Hey @TestBot help me", metadata: { threadTs: "9999.0001", channelId: "C456" } }, + { channelId: "slack", conversationId: "C456" }, + ); + + // Now send in the same thread -- should skip the ownership HTTP call. + const result = await hooks.message_sending( + { content: "Sure!", metadata: { threadTs: "9999.0001", channelId: "C456" }, to: "C456" }, + { channelId: "slack", conversationId: "C456" }, + ); + + expect(result).toBeUndefined(); + expect(globalThis.fetch).not.toHaveBeenCalled(); + }); + + it("ignores @-mentions on non-slack channels", async () => { + // Use a unique thread key so module-level state from other tests doesn't interfere. + await hooks.message_received( + { content: "Hey @TestBot", metadata: { threadTs: "7777.0001", channelId: "C999" } }, + { channelId: "discord", conversationId: "C999" }, + ); + + // The mention should not have been tracked, so sending should still call fetch. + vi.mocked(globalThis.fetch).mockResolvedValue( + new Response(JSON.stringify({ owner: "test-agent" }), { status: 200 }), + ); + + await hooks.message_sending( + { content: "Sure!", metadata: { threadTs: "7777.0001", channelId: "C999" }, to: "C999" }, + { channelId: "slack", conversationId: "C999" }, + ); + + expect(globalThis.fetch).toHaveBeenCalled(); + }); + + it("tracks bot user ID mentions via <@U999> syntax", async () => { + await hooks.message_received( + { content: "Hey <@U999> help", metadata: { threadTs: "8888.0001", channelId: "C789" } }, + { channelId: "slack", conversationId: "C789" }, + ); + + const result = await hooks.message_sending( + { content: "On it!", metadata: { threadTs: "8888.0001", channelId: "C789" }, to: "C789" }, + { channelId: "slack", conversationId: "C789" }, + ); + + expect(result).toBeUndefined(); + expect(globalThis.fetch).not.toHaveBeenCalled(); + }); + }); +}); diff --git a/extensions/thread-ownership/index.ts b/extensions/thread-ownership/index.ts new file mode 100644 index 00000000000..3db1ea94ff4 --- /dev/null +++ b/extensions/thread-ownership/index.ts @@ -0,0 +1,133 @@ +import type { OpenClawConfig, OpenClawPluginApi } from "openclaw/plugin-sdk"; + +type ThreadOwnershipConfig = { + forwarderUrl?: string; + abTestChannels?: string[]; +}; + +type AgentEntry = NonNullable["list"]>[number]; + +// In-memory set of {channel}:{thread} keys where this agent was @-mentioned. +// Entries expire after 5 minutes. +const mentionedThreads = new Map(); +const MENTION_TTL_MS = 5 * 60 * 1000; + +function cleanExpiredMentions(): void { + const now = Date.now(); + for (const [key, ts] of mentionedThreads) { + if (now - ts > MENTION_TTL_MS) { + mentionedThreads.delete(key); + } + } +} + +function resolveOwnershipAgent(config: OpenClawConfig): { id: string; name: string } { + const list = Array.isArray(config.agents?.list) + ? config.agents.list.filter((entry): entry is AgentEntry => + Boolean(entry && typeof entry === "object"), + ) + : []; + const selected = list.find((entry) => entry.default === true) ?? list[0]; + + const id = + typeof selected?.id === "string" && selected.id.trim() ? selected.id.trim() : "unknown"; + const identityName = + typeof selected?.identity?.name === "string" ? selected.identity.name.trim() : ""; + const fallbackName = typeof selected?.name === "string" ? selected.name.trim() : ""; + const name = identityName || fallbackName; + + return { id, name }; +} + +export default function register(api: OpenClawPluginApi) { + const pluginCfg = (api.pluginConfig ?? {}) as ThreadOwnershipConfig; + const forwarderUrl = ( + pluginCfg.forwarderUrl ?? + process.env.SLACK_FORWARDER_URL ?? + "http://slack-forwarder:8750" + ).replace(/\/$/, ""); + + const abTestChannels = new Set( + pluginCfg.abTestChannels ?? + process.env.THREAD_OWNERSHIP_CHANNELS?.split(",").filter(Boolean) ?? + [], + ); + + const { id: agentId, name: agentName } = resolveOwnershipAgent(api.config); + const botUserId = process.env.SLACK_BOT_USER_ID ?? ""; + + // --------------------------------------------------------------------------- + // message_received: track @-mentions so the agent can reply even if it + // doesn't own the thread. + // --------------------------------------------------------------------------- + api.on("message_received", async (event, ctx) => { + if (ctx.channelId !== "slack") return; + + const text = event.content ?? ""; + const threadTs = (event.metadata?.threadTs as string) ?? ""; + const channelId = (event.metadata?.channelId as string) ?? ctx.conversationId ?? ""; + + if (!threadTs || !channelId) return; + + // Check if this agent was @-mentioned. + const mentioned = + (agentName && text.includes(`@${agentName}`)) || + (botUserId && text.includes(`<@${botUserId}>`)); + + if (mentioned) { + cleanExpiredMentions(); + mentionedThreads.set(`${channelId}:${threadTs}`, Date.now()); + } + }); + + // --------------------------------------------------------------------------- + // message_sending: check thread ownership before sending to Slack. + // Returns { cancel: true } if another agent owns the thread. + // --------------------------------------------------------------------------- + api.on("message_sending", async (event, ctx) => { + if (ctx.channelId !== "slack") return; + + const threadTs = (event.metadata?.threadTs as string) ?? ""; + const channelId = (event.metadata?.channelId as string) ?? event.to; + + // Top-level messages (no thread) are always allowed. + if (!threadTs) return; + + // Only enforce in A/B test channels (if set is empty, skip entirely). + if (abTestChannels.size > 0 && !abTestChannels.has(channelId)) return; + + // If this agent was @-mentioned in this thread recently, skip ownership check. + cleanExpiredMentions(); + if (mentionedThreads.has(`${channelId}:${threadTs}`)) return; + + // Try to claim ownership via the forwarder HTTP API. + try { + const resp = await fetch(`${forwarderUrl}/api/v1/ownership/${channelId}/${threadTs}`, { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ agent_id: agentId }), + signal: AbortSignal.timeout(3000), + }); + + if (resp.ok) { + // We own it (or just claimed it), proceed. + return; + } + + if (resp.status === 409) { + // Another agent owns this thread β€” cancel the send. + const body = (await resp.json()) as { owner?: string }; + api.logger.info?.( + `thread-ownership: cancelled send to ${channelId}:${threadTs} β€” owned by ${body.owner}`, + ); + return { cancel: true }; + } + + // Unexpected status β€” fail open. + api.logger.warn?.(`thread-ownership: unexpected status ${resp.status}, allowing send`); + } catch (err) { + // Network error β€” fail open. + api.logger.warn?.(`thread-ownership: ownership check failed (${String(err)}), allowing send`); + } + }); +} diff --git a/extensions/thread-ownership/openclaw.plugin.json b/extensions/thread-ownership/openclaw.plugin.json new file mode 100644 index 00000000000..2e020bdadec --- /dev/null +++ b/extensions/thread-ownership/openclaw.plugin.json @@ -0,0 +1,28 @@ +{ + "id": "thread-ownership", + "name": "Thread Ownership", + "description": "Prevents multiple agents from responding in the same Slack thread. Uses HTTP calls to the slack-forwarder ownership API.", + "configSchema": { + "type": "object", + "additionalProperties": false, + "properties": { + "forwarderUrl": { + "type": "string" + }, + "abTestChannels": { + "type": "array", + "items": { "type": "string" } + } + } + }, + "uiHints": { + "forwarderUrl": { + "label": "Forwarder URL", + "help": "Base URL of the slack-forwarder ownership API (default: http://slack-forwarder:8750)" + }, + "abTestChannels": { + "label": "A/B Test Channels", + "help": "Slack channel IDs where thread ownership is enforced" + } + } +} diff --git a/scripts/pr b/scripts/pr index 1ceb0bce0af..3c51a331b1c 100755 --- a/scripts/pr +++ b/scripts/pr @@ -2,6 +2,18 @@ set -euo pipefail +# If invoked from a linked worktree copy of this script, re-exec the canonical +# script from the repository root so behavior stays consistent across worktrees. +script_self="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/$(basename "${BASH_SOURCE[0]}")" +script_parent_dir="$(dirname "$script_self")" +if common_git_dir=$(git -C "$script_parent_dir" rev-parse --path-format=absolute --git-common-dir 2>/dev/null); then + canonical_repo_root="$(dirname "$common_git_dir")" + canonical_self="$canonical_repo_root/scripts/$(basename "${BASH_SOURCE[0]}")" + if [ "$script_self" != "$canonical_self" ] && [ -x "$canonical_self" ]; then + exec "$canonical_self" "$@" + fi +fi + usage() { cat </dev/null); then + (cd "$(dirname "$common_git_dir")" && pwd) + return + fi + + # Fallback for environments where git common-dir is unavailable. (cd "$script_dir/.." && pwd) } diff --git a/scripts/pr-merge b/scripts/pr-merge index 745d74d8854..728c8289d0a 100755 --- a/scripts/pr-merge +++ b/scripts/pr-merge @@ -2,6 +2,13 @@ set -euo pipefail script_dir="$(cd "$(dirname "$0")" && pwd)" +base="$script_dir/pr" +if common_git_dir=$(git -C "$script_dir" rev-parse --path-format=absolute --git-common-dir 2>/dev/null); then + canonical_base="$(dirname "$common_git_dir")/scripts/pr" + if [ -x "$canonical_base" ]; then + base="$canonical_base" + fi +fi usage() { cat </dev/null); then + canonical_base="$(dirname "$common_git_dir")/scripts/pr" + if [ -x "$canonical_base" ]; then + base="$canonical_base" + fi +fi case "$mode" in init) diff --git a/scripts/pr-review b/scripts/pr-review index 1376080e156..afd765a8469 100755 --- a/scripts/pr-review +++ b/scripts/pr-review @@ -1,3 +1,13 @@ #!/usr/bin/env bash set -euo pipefail -exec "$(cd "$(dirname "$0")" && pwd)/pr" review-init "$@" + +script_dir="$(cd "$(dirname "$0")" && pwd)" +base="$script_dir/pr" +if common_git_dir=$(git -C "$script_dir" rev-parse --path-format=absolute --git-common-dir 2>/dev/null); then + canonical_base="$(dirname "$common_git_dir")/scripts/pr" + if [ -x "$canonical_base" ]; then + base="$canonical_base" + fi +fi + +exec "$base" review-init "$@" diff --git a/scripts/recover-orphaned-processes.sh b/scripts/recover-orphaned-processes.sh new file mode 100755 index 00000000000..d37c5ea4c80 --- /dev/null +++ b/scripts/recover-orphaned-processes.sh @@ -0,0 +1,191 @@ +#!/usr/bin/env bash +# Scan for orphaned coding agent processes after a gateway restart. +# +# Background coding agents (Claude Code, Codex CLI) spawned by the gateway +# can outlive the session that started them when the gateway restarts. +# This script finds them and reports their state. +# +# Usage: +# recover-orphaned-processes.sh +# +# Output: JSON object with `orphaned` array and `ts` timestamp. +set -euo pipefail + +usage() { + cat <<'USAGE' +Usage: recover-orphaned-processes.sh + +Scans for likely orphaned coding agent processes and prints JSON. +USAGE +} + +if [ "${1:-}" = "--help" ] || [ "${1:-}" = "-h" ]; then + usage + exit 0 +fi + +if [ "$#" -gt 0 ]; then + usage >&2 + exit 2 +fi + +if ! command -v node &>/dev/null; then + _ts="unknown" + command -v date &>/dev/null && _ts="$(date -u +%Y-%m-%dT%H:%M:%SZ 2>/dev/null)" || true + [ -z "$_ts" ] && _ts="unknown" + printf '{"error":"node not found on PATH","orphaned":[],"ts":"%s"}\n' "$_ts" + exit 0 +fi + +node <<'NODE' +const { execFileSync } = require("node:child_process"); +const fs = require("node:fs"); + +let username = process.env.USER || process.env.LOGNAME || ""; + +if (username && !/^[a-zA-Z0-9._-]+$/.test(username)) { + username = ""; +} + +function runFile(file, args) { + try { + return execFileSync(file, args, { + encoding: "utf8", + stdio: ["ignore", "pipe", "ignore"], + }); + } catch (err) { + if (err && typeof err.stdout === "string") { + return err.stdout; + } + if (err && err.stdout && Buffer.isBuffer(err.stdout)) { + return err.stdout.toString("utf8"); + } + return ""; + } +} + +function resolveStarted(pid) { + const started = runFile("ps", ["-o", "lstart=", "-p", String(pid)]).trim(); + return started.length > 0 ? started : "unknown"; +} + +function resolveCwd(pid) { + if (process.platform === "linux") { + try { + return fs.readlinkSync(`/proc/${pid}/cwd`); + } catch { + return "unknown"; + } + } + const lsof = runFile("lsof", ["-a", "-d", "cwd", "-p", String(pid), "-Fn"]); + const match = lsof.match(/^n(.+)$/m); + return match ? match[1] : "unknown"; +} + +function sanitizeCommand(cmd) { + // Avoid leaking obvious secrets when this diagnostic output is shared. + return cmd + .replace( + /(--(?:token|api[-_]?key|password|secret|authorization)\s+)([^\s]+)/gi, + "$1", + ) + .replace( + /((?:token|api[-_]?key|password|secret|authorization)=)([^\s]+)/gi, + "$1", + ) + .replace(/(Bearer\s+)[A-Za-z0-9._~+/=-]+/g, "$1"); +} + +// Pre-filter candidate PIDs using pgrep to avoid scanning all processes. +// Only falls back to a full ps scan when pgrep is genuinely unavailable +// (ENOENT), not when it simply finds no matches (exit code 1). +let pgrepUnavailable = false; +const pgrepResult = (() => { + const args = + username.length > 0 + ? ["-u", username, "-f", "codex|claude"] + : ["-f", "codex|claude"]; + try { + return execFileSync("pgrep", args, { + encoding: "utf8", + stdio: ["ignore", "pipe", "ignore"], + }); + } catch (err) { + if (err && err.code === "ENOENT") { + pgrepUnavailable = true; + return ""; + } + // pgrep exit code 1 = no matches β€” return stdout (empty) + if (err && typeof err.stdout === "string") return err.stdout; + return ""; + } +})(); + +const candidatePids = pgrepResult + .split("\n") + .map((s) => s.trim()) + .filter((s) => s.length > 0 && /^\d+$/.test(s)); + +let lines; +if (candidatePids.length > 0) { + // Fetch command info only for candidate PIDs. + lines = runFile("ps", ["-o", "pid=,command=", "-p", candidatePids.join(",")]).split("\n"); +} else if (pgrepUnavailable && username.length > 0) { + // pgrep not installed β€” fall back to user-scoped ps scan. + lines = runFile("ps", ["-U", username, "-o", "pid=,command="]).split("\n"); +} else if (pgrepUnavailable) { + // pgrep not installed and no username β€” full scan as last resort. + lines = runFile("ps", ["-axo", "pid=,command="]).split("\n"); +} else { + // pgrep ran successfully but found no matches β€” no orphans. + lines = []; +} + +const includePattern = /codex|claude/i; + +const excludePatterns = [ + /openclaw-gateway/i, + /signal-cli/i, + /node_modules\/\.bin\/openclaw/i, + /recover-orphaned-processes\.sh/i, +]; + +const orphaned = []; + +for (const rawLine of lines) { + const line = rawLine.trim(); + if (!line) { + continue; + } + const match = line.match(/^(\d+)\s+(.+)$/); + if (!match) { + continue; + } + + const pid = Number(match[1]); + const cmd = match[2]; + if (!Number.isInteger(pid) || pid <= 0 || pid === process.pid) { + continue; + } + if (!includePattern.test(cmd)) { + continue; + } + if (excludePatterns.some((pattern) => pattern.test(cmd))) { + continue; + } + + orphaned.push({ + pid, + cmd: sanitizeCommand(cmd), + cwd: resolveCwd(pid), + started: resolveStarted(pid), + }); +} + +process.stdout.write( + JSON.stringify({ + orphaned, + ts: new Date().toISOString(), + }) + "\n", +); +NODE diff --git a/scripts/run-node.mjs b/scripts/run-node.mjs index e02720a14fe..d06f70e5560 100644 --- a/scripts/run-node.mjs +++ b/scripts/run-node.mjs @@ -1,5 +1,5 @@ #!/usr/bin/env node -import { spawn } from "node:child_process"; +import { spawn, spawnSync } from "node:child_process"; import fs from "node:fs"; import path from "node:path"; import process from "node:process"; @@ -15,6 +15,7 @@ const distEntry = path.join(distRoot, "/entry.js"); const buildStampPath = path.join(distRoot, ".buildstamp"); const srcRoot = path.join(cwd, "src"); const configFiles = [path.join(cwd, "tsconfig.json"), path.join(cwd, "package.json")]; +const gitWatchedPaths = ["src", "tsconfig.json", "package.json"]; const statMtime = (filePath) => { try { @@ -74,12 +75,70 @@ const findLatestMtime = (dirPath, shouldSkip) => { return latest; }; +const runGit = (args) => { + try { + const result = spawnSync("git", args, { + cwd, + encoding: "utf8", + stdio: ["ignore", "pipe", "ignore"], + }); + if (result.status !== 0) { + return null; + } + return (result.stdout ?? "").trim(); + } catch { + return null; + } +}; + +const resolveGitHead = () => { + const head = runGit(["rev-parse", "HEAD"]); + return head || null; +}; + +const hasDirtySourceTree = () => { + const output = runGit([ + "status", + "--porcelain", + "--untracked-files=normal", + "--", + ...gitWatchedPaths, + ]); + if (output === null) { + return null; + } + return output.length > 0; +}; + +const readBuildStamp = () => { + const mtime = statMtime(buildStampPath); + if (mtime == null) { + return { mtime: null, head: null }; + } + try { + const raw = fs.readFileSync(buildStampPath, "utf8").trim(); + if (!raw.startsWith("{")) { + return { mtime, head: null }; + } + const parsed = JSON.parse(raw); + const head = typeof parsed?.head === "string" && parsed.head.trim() ? parsed.head.trim() : null; + return { mtime, head }; + } catch { + return { mtime, head: null }; + } +}; + +const hasSourceMtimeChanged = (stampMtime) => { + const srcMtime = findLatestMtime(srcRoot, isExcludedSource); + return srcMtime != null && srcMtime > stampMtime; +}; + const shouldBuild = () => { if (env.OPENCLAW_FORCE_BUILD === "1") { return true; } - const stampMtime = statMtime(buildStampPath); - if (stampMtime == null) { + const stamp = readBuildStamp(); + if (stamp.mtime == null) { return true; } if (statMtime(distEntry) == null) { @@ -88,13 +147,29 @@ const shouldBuild = () => { for (const filePath of configFiles) { const mtime = statMtime(filePath); - if (mtime != null && mtime > stampMtime) { + if (mtime != null && mtime > stamp.mtime) { return true; } } - const srcMtime = findLatestMtime(srcRoot, isExcludedSource); - if (srcMtime != null && srcMtime > stampMtime) { + const currentHead = resolveGitHead(); + if (currentHead && !stamp.head) { + return hasSourceMtimeChanged(stamp.mtime); + } + if (currentHead && stamp.head && currentHead !== stamp.head) { + return hasSourceMtimeChanged(stamp.mtime); + } + if (currentHead) { + const dirty = hasDirtySourceTree(); + if (dirty === true) { + return true; + } + if (dirty === false) { + return false; + } + } + + if (hasSourceMtimeChanged(stamp.mtime)) { return true; } return false; @@ -125,7 +200,11 @@ const runNode = () => { const writeBuildStamp = () => { try { fs.mkdirSync(distRoot, { recursive: true }); - fs.writeFileSync(buildStampPath, `${Date.now()}\n`); + const stamp = { + builtAt: Date.now(), + head: resolveGitHead(), + }; + fs.writeFileSync(buildStampPath, `${JSON.stringify(stamp)}\n`); } catch (error) { // Best-effort stamp; still allow the runner to start. logRunner(`Failed to write build stamp: ${error?.message ?? "unknown error"}`); diff --git a/scripts/test-parallel.mjs b/scripts/test-parallel.mjs index 3483b058c91..3dce3ce8d49 100644 --- a/scripts/test-parallel.mjs +++ b/scripts/test-parallel.mjs @@ -20,7 +20,6 @@ const unitIsolatedFiles = [ "src/browser/server.post-tabs-open-profile-unknown-returns-404.test.ts", "src/browser/server.agent-contract-snapshot-endpoints.test.ts", "src/browser/server.agent-contract-form-layout-act-commands.test.ts", - "src/browser/server.serves-status-starts-browser-requested.test.ts", "src/browser/server.skips-default-maxchars-explicitly-set-zero.test.ts", "src/browser/server.auth-token-gates-http.test.ts", "src/browser/server-context.remote-tab-ops.test.ts", diff --git a/scripts/watch-node.mjs b/scripts/watch-node.mjs index fc6d264677a..ad644b8727f 100644 --- a/scripts/watch-node.mjs +++ b/scripts/watch-node.mjs @@ -6,6 +6,12 @@ const args = process.argv.slice(2); const env = { ...process.env }; const cwd = process.cwd(); const compiler = "tsdown"; +const watchSession = `${Date.now()}-${process.pid}`; +env.OPENCLAW_WATCH_MODE = "1"; +env.OPENCLAW_WATCH_SESSION = watchSession; +if (args.length > 0) { + env.OPENCLAW_WATCH_COMMAND = args.join(" "); +} const initialBuild = spawnSync("pnpm", ["exec", compiler], { cwd, diff --git a/src/acp/server.ts b/src/acp/server.ts index 4a2c835b549..93acc4a523c 100644 --- a/src/acp/server.ts +++ b/src/acp/server.ts @@ -11,7 +11,7 @@ import { isMainModule } from "../infra/is-main.js"; import { GATEWAY_CLIENT_MODES, GATEWAY_CLIENT_NAMES } from "../utils/message-channel.js"; import { AcpGatewayAgent } from "./translator.js"; -export function serveAcpGateway(opts: AcpServerOptions = {}): void { +export function serveAcpGateway(opts: AcpServerOptions = {}): Promise { const cfg = loadConfig(); const connection = buildGatewayConnectionDetails({ config: cfg, @@ -34,6 +34,12 @@ export function serveAcpGateway(opts: AcpServerOptions = {}): void { auth.password; let agent: AcpGatewayAgent | null = null; + let onClosed!: () => void; + const closed = new Promise((resolve) => { + onClosed = resolve; + }); + let stopped = false; + const gateway = new GatewayClient({ url: connection.url, token: token || undefined, @@ -50,9 +56,29 @@ export function serveAcpGateway(opts: AcpServerOptions = {}): void { }, onClose: (code, reason) => { agent?.handleGatewayDisconnect(`${code}: ${reason}`); + // Resolve only on intentional shutdown (gateway.stop() sets closed + // which skips scheduleReconnect, then fires onClose). Transient + // disconnects are followed by automatic reconnect attempts. + if (stopped) { + onClosed(); + } }, }); + const shutdown = () => { + if (stopped) { + return; + } + stopped = true; + gateway.stop(); + // If no WebSocket is active (e.g. between reconnect attempts), + // gateway.stop() won't trigger onClose, so resolve directly. + onClosed(); + }; + + process.once("SIGINT", shutdown); + process.once("SIGTERM", shutdown); + const input = Writable.toWeb(process.stdout); const output = Readable.toWeb(process.stdin) as unknown as ReadableStream; const stream = ndJsonStream(input, output); @@ -64,6 +90,7 @@ export function serveAcpGateway(opts: AcpServerOptions = {}): void { }, stream); gateway.start(); + return closed; } function parseArgs(args: string[]): AcpServerOptions { @@ -140,5 +167,8 @@ Options: if (isMainModule({ currentFile: fileURLToPath(import.meta.url) })) { const opts = parseArgs(process.argv.slice(2)); - serveAcpGateway(opts); + serveAcpGateway(opts).catch((err) => { + console.error(String(err)); + process.exit(1); + }); } diff --git a/src/agents/bootstrap-files.ts b/src/agents/bootstrap-files.ts index 30e825171e9..0954cd40e15 100644 --- a/src/agents/bootstrap-files.ts +++ b/src/agents/bootstrap-files.ts @@ -30,6 +30,7 @@ export async function resolveBootstrapFilesForRun(params: { await loadWorkspaceBootstrapFiles(params.workspaceDir), sessionKey, ); + return applyBootstrapHookOverrides({ files: bootstrapFiles, workspaceDir: params.workspaceDir, diff --git a/src/agents/model-forward-compat.ts b/src/agents/model-forward-compat.ts new file mode 100644 index 00000000000..f92523e0296 --- /dev/null +++ b/src/agents/model-forward-compat.ts @@ -0,0 +1,240 @@ +import type { Api, Model } from "@mariozechner/pi-ai"; +import type { ModelRegistry } from "./pi-model-discovery.js"; +import { DEFAULT_CONTEXT_TOKENS } from "./defaults.js"; +import { normalizeModelCompat } from "./model-compat.js"; +import { normalizeProviderId } from "./model-selection.js"; + +const OPENAI_CODEX_GPT_53_MODEL_ID = "gpt-5.3-codex"; +const OPENAI_CODEX_TEMPLATE_MODEL_IDS = ["gpt-5.2-codex"] as const; + +const ANTHROPIC_OPUS_46_MODEL_ID = "claude-opus-4-6"; +const ANTHROPIC_OPUS_46_DOT_MODEL_ID = "claude-opus-4.6"; +const ANTHROPIC_OPUS_TEMPLATE_MODEL_IDS = ["claude-opus-4-5", "claude-opus-4.5"] as const; + +const ZAI_GLM5_MODEL_ID = "glm-5"; +const ZAI_GLM5_TEMPLATE_MODEL_IDS = ["glm-4.7"] as const; + +const ANTIGRAVITY_OPUS_46_MODEL_ID = "claude-opus-4-6"; +const ANTIGRAVITY_OPUS_46_DOT_MODEL_ID = "claude-opus-4.6"; +const ANTIGRAVITY_OPUS_TEMPLATE_MODEL_IDS = ["claude-opus-4-5", "claude-opus-4.5"] as const; +const ANTIGRAVITY_OPUS_46_THINKING_MODEL_ID = "claude-opus-4-6-thinking"; +const ANTIGRAVITY_OPUS_46_DOT_THINKING_MODEL_ID = "claude-opus-4.6-thinking"; +const ANTIGRAVITY_OPUS_THINKING_TEMPLATE_MODEL_IDS = [ + "claude-opus-4-5-thinking", + "claude-opus-4.5-thinking", +] as const; + +export const ANTIGRAVITY_OPUS_46_FORWARD_COMPAT_CANDIDATES = [ + { + id: ANTIGRAVITY_OPUS_46_THINKING_MODEL_ID, + templatePrefixes: [ + "google-antigravity/claude-opus-4-5-thinking", + "google-antigravity/claude-opus-4.5-thinking", + ], + }, + { + id: ANTIGRAVITY_OPUS_46_MODEL_ID, + templatePrefixes: ["google-antigravity/claude-opus-4-5", "google-antigravity/claude-opus-4.5"], + }, +] as const; + +function resolveOpenAICodexGpt53FallbackModel( + provider: string, + modelId: string, + modelRegistry: ModelRegistry, +): Model | undefined { + const normalizedProvider = normalizeProviderId(provider); + const trimmedModelId = modelId.trim(); + if (normalizedProvider !== "openai-codex") { + return undefined; + } + if (trimmedModelId.toLowerCase() !== OPENAI_CODEX_GPT_53_MODEL_ID) { + return undefined; + } + + for (const templateId of OPENAI_CODEX_TEMPLATE_MODEL_IDS) { + const template = modelRegistry.find(normalizedProvider, templateId) as Model | null; + if (!template) { + continue; + } + return normalizeModelCompat({ + ...template, + id: trimmedModelId, + name: trimmedModelId, + } as Model); + } + + return normalizeModelCompat({ + id: trimmedModelId, + name: trimmedModelId, + api: "openai-codex-responses", + provider: normalizedProvider, + baseUrl: "https://chatgpt.com/backend-api", + reasoning: true, + input: ["text", "image"], + cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 }, + contextWindow: DEFAULT_CONTEXT_TOKENS, + maxTokens: DEFAULT_CONTEXT_TOKENS, + } as Model); +} + +function resolveAnthropicOpus46ForwardCompatModel( + provider: string, + modelId: string, + modelRegistry: ModelRegistry, +): Model | undefined { + const normalizedProvider = normalizeProviderId(provider); + if (normalizedProvider !== "anthropic") { + return undefined; + } + + const trimmedModelId = modelId.trim(); + const lower = trimmedModelId.toLowerCase(); + const isOpus46 = + lower === ANTHROPIC_OPUS_46_MODEL_ID || + lower === ANTHROPIC_OPUS_46_DOT_MODEL_ID || + lower.startsWith(`${ANTHROPIC_OPUS_46_MODEL_ID}-`) || + lower.startsWith(`${ANTHROPIC_OPUS_46_DOT_MODEL_ID}-`); + if (!isOpus46) { + return undefined; + } + + const templateIds: string[] = []; + if (lower.startsWith(ANTHROPIC_OPUS_46_MODEL_ID)) { + templateIds.push(lower.replace(ANTHROPIC_OPUS_46_MODEL_ID, "claude-opus-4-5")); + } + if (lower.startsWith(ANTHROPIC_OPUS_46_DOT_MODEL_ID)) { + templateIds.push(lower.replace(ANTHROPIC_OPUS_46_DOT_MODEL_ID, "claude-opus-4.5")); + } + templateIds.push(...ANTHROPIC_OPUS_TEMPLATE_MODEL_IDS); + + for (const templateId of [...new Set(templateIds)].filter(Boolean)) { + const template = modelRegistry.find(normalizedProvider, templateId) as Model | null; + if (!template) { + continue; + } + return normalizeModelCompat({ + ...template, + id: trimmedModelId, + name: trimmedModelId, + } as Model); + } + + return undefined; +} + +// Z.ai's GLM-5 may not be present in pi-ai's built-in model catalog yet. +// When a user configures zai/glm-5 without a models.json entry, clone glm-4.7 as a forward-compat fallback. +function resolveZaiGlm5ForwardCompatModel( + provider: string, + modelId: string, + modelRegistry: ModelRegistry, +): Model | undefined { + if (normalizeProviderId(provider) !== "zai") { + return undefined; + } + const trimmed = modelId.trim(); + const lower = trimmed.toLowerCase(); + if (lower !== ZAI_GLM5_MODEL_ID && !lower.startsWith(`${ZAI_GLM5_MODEL_ID}-`)) { + return undefined; + } + + for (const templateId of ZAI_GLM5_TEMPLATE_MODEL_IDS) { + const template = modelRegistry.find("zai", templateId) as Model | null; + if (!template) { + continue; + } + return normalizeModelCompat({ + ...template, + id: trimmed, + name: trimmed, + reasoning: true, + } as Model); + } + + return normalizeModelCompat({ + id: trimmed, + name: trimmed, + api: "openai-completions", + provider: "zai", + reasoning: true, + input: ["text"], + cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 }, + contextWindow: DEFAULT_CONTEXT_TOKENS, + maxTokens: DEFAULT_CONTEXT_TOKENS, + } as Model); +} + +function resolveAntigravityOpus46ForwardCompatModel( + provider: string, + modelId: string, + modelRegistry: ModelRegistry, +): Model | undefined { + const normalizedProvider = normalizeProviderId(provider); + if (normalizedProvider !== "google-antigravity") { + return undefined; + } + + const trimmedModelId = modelId.trim(); + const lower = trimmedModelId.toLowerCase(); + const isOpus46 = + lower === ANTIGRAVITY_OPUS_46_MODEL_ID || + lower === ANTIGRAVITY_OPUS_46_DOT_MODEL_ID || + lower.startsWith(`${ANTIGRAVITY_OPUS_46_MODEL_ID}-`) || + lower.startsWith(`${ANTIGRAVITY_OPUS_46_DOT_MODEL_ID}-`); + const isOpus46Thinking = + lower === ANTIGRAVITY_OPUS_46_THINKING_MODEL_ID || + lower === ANTIGRAVITY_OPUS_46_DOT_THINKING_MODEL_ID || + lower.startsWith(`${ANTIGRAVITY_OPUS_46_THINKING_MODEL_ID}-`) || + lower.startsWith(`${ANTIGRAVITY_OPUS_46_DOT_THINKING_MODEL_ID}-`); + if (!isOpus46 && !isOpus46Thinking) { + return undefined; + } + + const templateIds: string[] = []; + if (lower.startsWith(ANTIGRAVITY_OPUS_46_MODEL_ID)) { + templateIds.push(lower.replace(ANTIGRAVITY_OPUS_46_MODEL_ID, "claude-opus-4-5")); + } + if (lower.startsWith(ANTIGRAVITY_OPUS_46_DOT_MODEL_ID)) { + templateIds.push(lower.replace(ANTIGRAVITY_OPUS_46_DOT_MODEL_ID, "claude-opus-4.5")); + } + if (lower.startsWith(ANTIGRAVITY_OPUS_46_THINKING_MODEL_ID)) { + templateIds.push( + lower.replace(ANTIGRAVITY_OPUS_46_THINKING_MODEL_ID, "claude-opus-4-5-thinking"), + ); + } + if (lower.startsWith(ANTIGRAVITY_OPUS_46_DOT_THINKING_MODEL_ID)) { + templateIds.push( + lower.replace(ANTIGRAVITY_OPUS_46_DOT_THINKING_MODEL_ID, "claude-opus-4.5-thinking"), + ); + } + templateIds.push(...ANTIGRAVITY_OPUS_TEMPLATE_MODEL_IDS); + templateIds.push(...ANTIGRAVITY_OPUS_THINKING_TEMPLATE_MODEL_IDS); + + for (const templateId of [...new Set(templateIds)].filter(Boolean)) { + const template = modelRegistry.find(normalizedProvider, templateId) as Model | null; + if (!template) { + continue; + } + return normalizeModelCompat({ + ...template, + id: trimmedModelId, + name: trimmedModelId, + } as Model); + } + + return undefined; +} + +export function resolveForwardCompatModel( + provider: string, + modelId: string, + modelRegistry: ModelRegistry, +): Model | undefined { + return ( + resolveOpenAICodexGpt53FallbackModel(provider, modelId, modelRegistry) ?? + resolveAnthropicOpus46ForwardCompatModel(provider, modelId, modelRegistry) ?? + resolveZaiGlm5ForwardCompatModel(provider, modelId, modelRegistry) ?? + resolveAntigravityOpus46ForwardCompatModel(provider, modelId, modelRegistry) + ); +} diff --git a/src/agents/models-config.providers.ollama.e2e.test.ts b/src/agents/models-config.providers.ollama.e2e.test.ts index 3b9624a8eb6..263ef5574d4 100644 --- a/src/agents/models-config.providers.ollama.e2e.test.ts +++ b/src/agents/models-config.providers.ollama.e2e.test.ts @@ -29,25 +29,20 @@ describe("Ollama provider", () => { const agentDir = mkdtempSync(join(tmpdir(), "openclaw-test-")); const providers = await resolveImplicitProviders({ agentDir }); - // Ollama requires explicit configuration via OLLAMA_API_KEY env var or profile expect(providers?.ollama).toBeUndefined(); }); - it("should disable streaming by default for Ollama models", async () => { + it("should use native ollama api type", async () => { const agentDir = mkdtempSync(join(tmpdir(), "openclaw-test-")); process.env.OLLAMA_API_KEY = "test-key"; try { const providers = await resolveImplicitProviders({ agentDir }); - // Provider should be defined with OLLAMA_API_KEY set expect(providers?.ollama).toBeDefined(); expect(providers?.ollama?.apiKey).toBe("OLLAMA_API_KEY"); - - // Note: discoverOllamaModels() returns empty array in test environments (VITEST env var check) - // so we can't test the actual model discovery here. The streaming: false setting - // is applied in the model mapping within discoverOllamaModels(). - // The configuration structure itself is validated by TypeScript and the Zod schema. + expect(providers?.ollama?.api).toBe("ollama"); + expect(providers?.ollama?.baseUrl).toBe("http://127.0.0.1:11434"); } finally { delete process.env.OLLAMA_API_KEY; } @@ -69,15 +64,14 @@ describe("Ollama provider", () => { }, }); - expect(providers?.ollama?.baseUrl).toBe("http://192.168.20.14:11434/v1"); + // Native API strips /v1 suffix via resolveOllamaApiBase() + expect(providers?.ollama?.baseUrl).toBe("http://192.168.20.14:11434"); } finally { delete process.env.OLLAMA_API_KEY; } }); - it("should have correct model structure with streaming disabled (unit test)", () => { - // This test directly verifies the model configuration structure - // since discoverOllamaModels() returns empty array in test mode + it("should have correct model structure without streaming override", () => { const mockOllamaModel = { id: "llama3.3:latest", name: "llama3.3:latest", @@ -86,13 +80,9 @@ describe("Ollama provider", () => { cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 }, contextWindow: 128000, maxTokens: 8192, - params: { - streaming: false, - }, }; - // Verify the model structure matches what discoverOllamaModels() would return - expect(mockOllamaModel.params?.streaming).toBe(false); - expect(mockOllamaModel.params).toHaveProperty("streaming"); + // Native Ollama provider does not need streaming: false workaround + expect(mockOllamaModel).not.toHaveProperty("params"); }); }); diff --git a/src/agents/models-config.providers.ts b/src/agents/models-config.providers.ts index aa6adfd434a..32835dc0f64 100644 --- a/src/agents/models-config.providers.ts +++ b/src/agents/models-config.providers.ts @@ -17,6 +17,7 @@ import { buildHuggingfaceModelDefinition, } from "./huggingface-models.js"; import { resolveAwsSdkEnvVarName, resolveEnvApiKey } from "./model-auth.js"; +import { OLLAMA_NATIVE_BASE_URL } from "./ollama-stream.js"; import { buildSyntheticModelDefinition, SYNTHETIC_BASE_URL, @@ -79,8 +80,8 @@ const QWEN_PORTAL_DEFAULT_COST = { cacheWrite: 0, }; -const OLLAMA_BASE_URL = "http://127.0.0.1:11434/v1"; -const OLLAMA_API_BASE_URL = "http://127.0.0.1:11434"; +const OLLAMA_BASE_URL = OLLAMA_NATIVE_BASE_URL; +const OLLAMA_API_BASE_URL = OLLAMA_BASE_URL; const OLLAMA_DEFAULT_CONTEXT_WINDOW = 128000; const OLLAMA_DEFAULT_MAX_TOKENS = 8192; const OLLAMA_DEFAULT_COST = { @@ -180,11 +181,6 @@ async function discoverOllamaModels(baseUrl?: string): Promise { async function buildOllamaProvider(configuredBaseUrl?: string): Promise { const models = await discoverOllamaModels(configuredBaseUrl); return { - baseUrl: configuredBaseUrl ?? OLLAMA_BASE_URL, - api: "openai-completions", + baseUrl: resolveOllamaApiBase(configuredBaseUrl), + api: "ollama", models, }; } diff --git a/src/agents/ollama-stream.test.ts b/src/agents/ollama-stream.test.ts new file mode 100644 index 00000000000..1589f2f25c8 --- /dev/null +++ b/src/agents/ollama-stream.test.ts @@ -0,0 +1,290 @@ +import { describe, expect, it, vi } from "vitest"; +import { + createOllamaStreamFn, + convertToOllamaMessages, + buildAssistantMessage, + parseNdjsonStream, +} from "./ollama-stream.js"; + +describe("convertToOllamaMessages", () => { + it("converts user text messages", () => { + const messages = [{ role: "user", content: "hello" }]; + const result = convertToOllamaMessages(messages); + expect(result).toEqual([{ role: "user", content: "hello" }]); + }); + + it("converts user messages with content parts", () => { + const messages = [ + { + role: "user", + content: [ + { type: "text", text: "describe this" }, + { type: "image", data: "base64data" }, + ], + }, + ]; + const result = convertToOllamaMessages(messages); + expect(result).toEqual([{ role: "user", content: "describe this", images: ["base64data"] }]); + }); + + it("prepends system message when provided", () => { + const messages = [{ role: "user", content: "hello" }]; + const result = convertToOllamaMessages(messages, "You are helpful."); + expect(result[0]).toEqual({ role: "system", content: "You are helpful." }); + expect(result[1]).toEqual({ role: "user", content: "hello" }); + }); + + it("converts assistant messages with toolCall content blocks", () => { + const messages = [ + { + role: "assistant", + content: [ + { type: "text", text: "Let me check." }, + { type: "toolCall", id: "call_1", name: "bash", arguments: { command: "ls" } }, + ], + }, + ]; + const result = convertToOllamaMessages(messages); + expect(result[0].role).toBe("assistant"); + expect(result[0].content).toBe("Let me check."); + expect(result[0].tool_calls).toEqual([ + { function: { name: "bash", arguments: { command: "ls" } } }, + ]); + }); + + it("converts tool result messages with 'tool' role", () => { + const messages = [{ role: "tool", content: "file1.txt\nfile2.txt" }]; + const result = convertToOllamaMessages(messages); + expect(result).toEqual([{ role: "tool", content: "file1.txt\nfile2.txt" }]); + }); + + it("converts SDK 'toolResult' role to Ollama 'tool' role", () => { + const messages = [{ role: "toolResult", content: "command output here" }]; + const result = convertToOllamaMessages(messages); + expect(result).toEqual([{ role: "tool", content: "command output here" }]); + }); + + it("includes tool_name from SDK toolResult messages", () => { + const messages = [{ role: "toolResult", content: "file contents here", toolName: "read" }]; + const result = convertToOllamaMessages(messages); + expect(result).toEqual([{ role: "tool", content: "file contents here", tool_name: "read" }]); + }); + + it("omits tool_name when not provided in toolResult", () => { + const messages = [{ role: "toolResult", content: "output" }]; + const result = convertToOllamaMessages(messages); + expect(result).toEqual([{ role: "tool", content: "output" }]); + expect(result[0]).not.toHaveProperty("tool_name"); + }); + + it("handles empty messages array", () => { + const result = convertToOllamaMessages([]); + expect(result).toEqual([]); + }); +}); + +describe("buildAssistantMessage", () => { + const modelInfo = { api: "ollama", provider: "ollama", id: "qwen3:32b" }; + + it("builds text-only response", () => { + const response = { + model: "qwen3:32b", + created_at: "2026-01-01T00:00:00Z", + message: { role: "assistant" as const, content: "Hello!" }, + done: true, + prompt_eval_count: 10, + eval_count: 5, + }; + const result = buildAssistantMessage(response, modelInfo); + expect(result.role).toBe("assistant"); + expect(result.content).toEqual([{ type: "text", text: "Hello!" }]); + expect(result.stopReason).toBe("stop"); + expect(result.usage.input).toBe(10); + expect(result.usage.output).toBe(5); + expect(result.usage.totalTokens).toBe(15); + }); + + it("builds response with tool calls", () => { + const response = { + model: "qwen3:32b", + created_at: "2026-01-01T00:00:00Z", + message: { + role: "assistant" as const, + content: "", + tool_calls: [{ function: { name: "bash", arguments: { command: "ls -la" } } }], + }, + done: true, + prompt_eval_count: 20, + eval_count: 10, + }; + const result = buildAssistantMessage(response, modelInfo); + expect(result.stopReason).toBe("toolUse"); + expect(result.content.length).toBe(1); // toolCall only (empty content is skipped) + expect(result.content[0].type).toBe("toolCall"); + const toolCall = result.content[0] as { + type: "toolCall"; + id: string; + name: string; + arguments: Record; + }; + expect(toolCall.name).toBe("bash"); + expect(toolCall.arguments).toEqual({ command: "ls -la" }); + expect(toolCall.id).toMatch(/^ollama_call_[0-9a-f-]{36}$/); + }); + + it("sets all costs to zero for local models", () => { + const response = { + model: "qwen3:32b", + created_at: "2026-01-01T00:00:00Z", + message: { role: "assistant" as const, content: "ok" }, + done: true, + }; + const result = buildAssistantMessage(response, modelInfo); + expect(result.usage.cost).toEqual({ + input: 0, + output: 0, + cacheRead: 0, + cacheWrite: 0, + total: 0, + }); + }); +}); + +// Helper: build a ReadableStreamDefaultReader from NDJSON lines +function mockNdjsonReader(lines: string[]): ReadableStreamDefaultReader { + const encoder = new TextEncoder(); + const payload = lines.join("\n") + "\n"; + let consumed = false; + return { + read: async () => { + if (consumed) { + return { done: true as const, value: undefined }; + } + consumed = true; + return { done: false as const, value: encoder.encode(payload) }; + }, + releaseLock: () => {}, + cancel: async () => {}, + closed: Promise.resolve(undefined), + } as unknown as ReadableStreamDefaultReader; +} + +describe("parseNdjsonStream", () => { + it("parses text-only streaming chunks", async () => { + const reader = mockNdjsonReader([ + '{"model":"m","created_at":"t","message":{"role":"assistant","content":"Hello"},"done":false}', + '{"model":"m","created_at":"t","message":{"role":"assistant","content":" world"},"done":false}', + '{"model":"m","created_at":"t","message":{"role":"assistant","content":""},"done":true,"prompt_eval_count":5,"eval_count":2}', + ]); + const chunks = []; + for await (const chunk of parseNdjsonStream(reader)) { + chunks.push(chunk); + } + expect(chunks).toHaveLength(3); + expect(chunks[0].message.content).toBe("Hello"); + expect(chunks[1].message.content).toBe(" world"); + expect(chunks[2].done).toBe(true); + }); + + it("parses tool_calls from intermediate chunk (not final)", async () => { + // Ollama sends tool_calls in done:false chunk, final done:true has no tool_calls + const reader = mockNdjsonReader([ + '{"model":"m","created_at":"t","message":{"role":"assistant","content":"","tool_calls":[{"function":{"name":"bash","arguments":{"command":"ls"}}}]},"done":false}', + '{"model":"m","created_at":"t","message":{"role":"assistant","content":""},"done":true,"prompt_eval_count":10,"eval_count":5}', + ]); + const chunks = []; + for await (const chunk of parseNdjsonStream(reader)) { + chunks.push(chunk); + } + expect(chunks).toHaveLength(2); + expect(chunks[0].done).toBe(false); + expect(chunks[0].message.tool_calls).toHaveLength(1); + expect(chunks[0].message.tool_calls![0].function.name).toBe("bash"); + expect(chunks[1].done).toBe(true); + expect(chunks[1].message.tool_calls).toBeUndefined(); + }); + + it("accumulates tool_calls across multiple intermediate chunks", async () => { + const reader = mockNdjsonReader([ + '{"model":"m","created_at":"t","message":{"role":"assistant","content":"","tool_calls":[{"function":{"name":"read","arguments":{"path":"/tmp/a"}}}]},"done":false}', + '{"model":"m","created_at":"t","message":{"role":"assistant","content":"","tool_calls":[{"function":{"name":"bash","arguments":{"command":"ls"}}}]},"done":false}', + '{"model":"m","created_at":"t","message":{"role":"assistant","content":""},"done":true}', + ]); + + // Simulate the accumulation logic from createOllamaStreamFn + const accumulatedToolCalls: Array<{ + function: { name: string; arguments: Record }; + }> = []; + const chunks = []; + for await (const chunk of parseNdjsonStream(reader)) { + chunks.push(chunk); + if (chunk.message?.tool_calls) { + accumulatedToolCalls.push(...chunk.message.tool_calls); + } + } + expect(accumulatedToolCalls).toHaveLength(2); + expect(accumulatedToolCalls[0].function.name).toBe("read"); + expect(accumulatedToolCalls[1].function.name).toBe("bash"); + // Final done:true chunk has no tool_calls + expect(chunks[2].message.tool_calls).toBeUndefined(); + }); +}); + +describe("createOllamaStreamFn", () => { + it("normalizes /v1 baseUrl and maps maxTokens + signal", async () => { + const originalFetch = globalThis.fetch; + const fetchMock = vi.fn(async () => { + const payload = [ + '{"model":"m","created_at":"t","message":{"role":"assistant","content":"ok"},"done":false}', + '{"model":"m","created_at":"t","message":{"role":"assistant","content":""},"done":true,"prompt_eval_count":1,"eval_count":1}', + ].join("\n"); + return new Response(`${payload}\n`, { + status: 200, + headers: { "Content-Type": "application/x-ndjson" }, + }); + }); + globalThis.fetch = fetchMock as unknown as typeof fetch; + + try { + const streamFn = createOllamaStreamFn("http://ollama-host:11434/v1/"); + const signal = new AbortController().signal; + const stream = streamFn( + { + id: "qwen3:32b", + api: "ollama", + provider: "custom-ollama", + contextWindow: 131072, + } as unknown as Parameters[0], + { + messages: [{ role: "user", content: "hello" }], + } as unknown as Parameters[1], + { + maxTokens: 123, + signal, + } as unknown as Parameters[2], + ); + + const events = []; + for await (const event of stream) { + events.push(event); + } + expect(events.at(-1)?.type).toBe("done"); + + expect(fetchMock).toHaveBeenCalledTimes(1); + const [url, requestInit] = fetchMock.mock.calls[0] as [string, RequestInit]; + expect(url).toBe("http://ollama-host:11434/api/chat"); + expect(requestInit.signal).toBe(signal); + if (typeof requestInit.body !== "string") { + throw new Error("Expected string request body"); + } + + const requestBody = JSON.parse(requestInit.body) as { + options: { num_ctx?: number; num_predict?: number }; + }; + expect(requestBody.options.num_ctx).toBe(131072); + expect(requestBody.options.num_predict).toBe(123); + } finally { + globalThis.fetch = originalFetch; + } + }); +}); diff --git a/src/agents/ollama-stream.ts b/src/agents/ollama-stream.ts new file mode 100644 index 00000000000..76029e67cea --- /dev/null +++ b/src/agents/ollama-stream.ts @@ -0,0 +1,419 @@ +import type { StreamFn } from "@mariozechner/pi-agent-core"; +import type { + AssistantMessage, + StopReason, + TextContent, + ToolCall, + Tool, + Usage, +} from "@mariozechner/pi-ai"; +import { createAssistantMessageEventStream } from "@mariozechner/pi-ai"; +import { randomUUID } from "node:crypto"; + +export const OLLAMA_NATIVE_BASE_URL = "http://127.0.0.1:11434"; + +// ── Ollama /api/chat request types ────────────────────────────────────────── + +interface OllamaChatRequest { + model: string; + messages: OllamaChatMessage[]; + stream: boolean; + tools?: OllamaTool[]; + options?: Record; +} + +interface OllamaChatMessage { + role: "system" | "user" | "assistant" | "tool"; + content: string; + images?: string[]; + tool_calls?: OllamaToolCall[]; + tool_name?: string; +} + +interface OllamaTool { + type: "function"; + function: { + name: string; + description: string; + parameters: Record; + }; +} + +interface OllamaToolCall { + function: { + name: string; + arguments: Record; + }; +} + +// ── Ollama /api/chat response types ───────────────────────────────────────── + +interface OllamaChatResponse { + model: string; + created_at: string; + message: { + role: "assistant"; + content: string; + tool_calls?: OllamaToolCall[]; + }; + done: boolean; + done_reason?: string; + total_duration?: number; + load_duration?: number; + prompt_eval_count?: number; + prompt_eval_duration?: number; + eval_count?: number; + eval_duration?: number; +} + +// ── Message conversion ────────────────────────────────────────────────────── + +type InputContentPart = + | { type: "text"; text: string } + | { type: "image"; data: string } + | { type: "toolCall"; id: string; name: string; arguments: Record } + | { type: "tool_use"; id: string; name: string; input: Record }; + +function extractTextContent(content: unknown): string { + if (typeof content === "string") { + return content; + } + if (!Array.isArray(content)) { + return ""; + } + return (content as InputContentPart[]) + .filter((part): part is { type: "text"; text: string } => part.type === "text") + .map((part) => part.text) + .join(""); +} + +function extractOllamaImages(content: unknown): string[] { + if (!Array.isArray(content)) { + return []; + } + return (content as InputContentPart[]) + .filter((part): part is { type: "image"; data: string } => part.type === "image") + .map((part) => part.data); +} + +function extractToolCalls(content: unknown): OllamaToolCall[] { + if (!Array.isArray(content)) { + return []; + } + const parts = content as InputContentPart[]; + const result: OllamaToolCall[] = []; + for (const part of parts) { + if (part.type === "toolCall") { + result.push({ function: { name: part.name, arguments: part.arguments } }); + } else if (part.type === "tool_use") { + result.push({ function: { name: part.name, arguments: part.input } }); + } + } + return result; +} + +export function convertToOllamaMessages( + messages: Array<{ role: string; content: unknown }>, + system?: string, +): OllamaChatMessage[] { + const result: OllamaChatMessage[] = []; + + if (system) { + result.push({ role: "system", content: system }); + } + + for (const msg of messages) { + const { role } = msg; + + if (role === "user") { + const text = extractTextContent(msg.content); + const images = extractOllamaImages(msg.content); + result.push({ + role: "user", + content: text, + ...(images.length > 0 ? { images } : {}), + }); + } else if (role === "assistant") { + const text = extractTextContent(msg.content); + const toolCalls = extractToolCalls(msg.content); + result.push({ + role: "assistant", + content: text, + ...(toolCalls.length > 0 ? { tool_calls: toolCalls } : {}), + }); + } else if (role === "tool" || role === "toolResult") { + // SDK uses "toolResult" (camelCase) for tool result messages. + // Ollama API expects "tool" role with tool_name per the native spec. + const text = extractTextContent(msg.content); + const toolName = + typeof (msg as { toolName?: unknown }).toolName === "string" + ? (msg as { toolName?: string }).toolName + : undefined; + result.push({ + role: "tool", + content: text, + ...(toolName ? { tool_name: toolName } : {}), + }); + } + } + + return result; +} + +// ── Tool extraction ───────────────────────────────────────────────────────── + +function extractOllamaTools(tools: Tool[] | undefined): OllamaTool[] { + if (!tools || !Array.isArray(tools)) { + return []; + } + const result: OllamaTool[] = []; + for (const tool of tools) { + if (typeof tool.name !== "string" || !tool.name) { + continue; + } + result.push({ + type: "function", + function: { + name: tool.name, + description: typeof tool.description === "string" ? tool.description : "", + parameters: (tool.parameters ?? {}) as Record, + }, + }); + } + return result; +} + +// ── Response conversion ───────────────────────────────────────────────────── + +export function buildAssistantMessage( + response: OllamaChatResponse, + modelInfo: { api: string; provider: string; id: string }, +): AssistantMessage { + const content: (TextContent | ToolCall)[] = []; + + if (response.message.content) { + content.push({ type: "text", text: response.message.content }); + } + + const toolCalls = response.message.tool_calls; + if (toolCalls && toolCalls.length > 0) { + for (const tc of toolCalls) { + content.push({ + type: "toolCall", + id: `ollama_call_${randomUUID()}`, + name: tc.function.name, + arguments: tc.function.arguments, + }); + } + } + + const hasToolCalls = toolCalls && toolCalls.length > 0; + const stopReason: StopReason = hasToolCalls ? "toolUse" : "stop"; + + const usage: Usage = { + input: response.prompt_eval_count ?? 0, + output: response.eval_count ?? 0, + cacheRead: 0, + cacheWrite: 0, + totalTokens: (response.prompt_eval_count ?? 0) + (response.eval_count ?? 0), + cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 }, + }; + + return { + role: "assistant", + content, + stopReason, + api: modelInfo.api, + provider: modelInfo.provider, + model: modelInfo.id, + usage, + timestamp: Date.now(), + }; +} + +// ── NDJSON streaming parser ───────────────────────────────────────────────── + +export async function* parseNdjsonStream( + reader: ReadableStreamDefaultReader, +): AsyncGenerator { + const decoder = new TextDecoder(); + let buffer = ""; + + while (true) { + const { done, value } = await reader.read(); + if (done) { + break; + } + buffer += decoder.decode(value, { stream: true }); + const lines = buffer.split("\n"); + buffer = lines.pop() ?? ""; + + for (const line of lines) { + const trimmed = line.trim(); + if (!trimmed) { + continue; + } + try { + yield JSON.parse(trimmed) as OllamaChatResponse; + } catch { + console.warn("[ollama-stream] Skipping malformed NDJSON line:", trimmed.slice(0, 120)); + } + } + } + + if (buffer.trim()) { + try { + yield JSON.parse(buffer.trim()) as OllamaChatResponse; + } catch { + console.warn( + "[ollama-stream] Skipping malformed trailing data:", + buffer.trim().slice(0, 120), + ); + } + } +} + +// ── Main StreamFn factory ─────────────────────────────────────────────────── + +function resolveOllamaChatUrl(baseUrl: string): string { + const trimmed = baseUrl.trim().replace(/\/+$/, ""); + const normalizedBase = trimmed.replace(/\/v1$/i, ""); + const apiBase = normalizedBase || OLLAMA_NATIVE_BASE_URL; + return `${apiBase}/api/chat`; +} + +export function createOllamaStreamFn(baseUrl: string): StreamFn { + const chatUrl = resolveOllamaChatUrl(baseUrl); + + return (model, context, options) => { + const stream = createAssistantMessageEventStream(); + + const run = async () => { + try { + const ollamaMessages = convertToOllamaMessages( + context.messages ?? [], + context.systemPrompt, + ); + + const ollamaTools = extractOllamaTools(context.tools); + + // Ollama defaults to num_ctx=4096 which is too small for large + // system prompts + many tool definitions. Use model's contextWindow. + const ollamaOptions: Record = { num_ctx: model.contextWindow ?? 65536 }; + if (typeof options?.temperature === "number") { + ollamaOptions.temperature = options.temperature; + } + if (typeof options?.maxTokens === "number") { + ollamaOptions.num_predict = options.maxTokens; + } + + const body: OllamaChatRequest = { + model: model.id, + messages: ollamaMessages, + stream: true, + ...(ollamaTools.length > 0 ? { tools: ollamaTools } : {}), + options: ollamaOptions, + }; + + const headers: Record = { + "Content-Type": "application/json", + ...options?.headers, + }; + if (options?.apiKey) { + headers.Authorization = `Bearer ${options.apiKey}`; + } + + const response = await fetch(chatUrl, { + method: "POST", + headers, + body: JSON.stringify(body), + signal: options?.signal, + }); + + if (!response.ok) { + const errorText = await response.text().catch(() => "unknown error"); + throw new Error(`Ollama API error ${response.status}: ${errorText}`); + } + + if (!response.body) { + throw new Error("Ollama API returned empty response body"); + } + + const reader = response.body.getReader(); + let accumulatedContent = ""; + const accumulatedToolCalls: OllamaToolCall[] = []; + let finalResponse: OllamaChatResponse | undefined; + + for await (const chunk of parseNdjsonStream(reader)) { + if (chunk.message?.content) { + accumulatedContent += chunk.message.content; + } + + // Ollama sends tool_calls in intermediate (done:false) chunks, + // NOT in the final done:true chunk. Collect from all chunks. + if (chunk.message?.tool_calls) { + accumulatedToolCalls.push(...chunk.message.tool_calls); + } + + if (chunk.done) { + finalResponse = chunk; + break; + } + } + + if (!finalResponse) { + throw new Error("Ollama API stream ended without a final response"); + } + + finalResponse.message.content = accumulatedContent; + if (accumulatedToolCalls.length > 0) { + finalResponse.message.tool_calls = accumulatedToolCalls; + } + + const assistantMessage = buildAssistantMessage(finalResponse, { + api: model.api, + provider: model.provider, + id: model.id, + }); + + const reason: Extract = + assistantMessage.stopReason === "toolUse" ? "toolUse" : "stop"; + + stream.push({ + type: "done", + reason, + message: assistantMessage, + }); + } catch (err) { + const errorMessage = err instanceof Error ? err.message : String(err); + stream.push({ + type: "error", + reason: "error", + error: { + role: "assistant" as const, + content: [], + stopReason: "error" as StopReason, + errorMessage, + api: model.api, + provider: model.provider, + model: model.id, + usage: { + input: 0, + output: 0, + cacheRead: 0, + cacheWrite: 0, + totalTokens: 0, + cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 }, + }, + timestamp: Date.now(), + }, + }); + } finally { + stream.end(); + } + }; + + queueMicrotask(() => void run()); + return stream; + }; +} diff --git a/src/agents/openai-responses.reasoning-replay.test.ts b/src/agents/openai-responses.reasoning-replay.test.ts index de4b10cd62d..2a94db7e3fd 100644 --- a/src/agents/openai-responses.reasoning-replay.test.ts +++ b/src/agents/openai-responses.reasoning-replay.test.ts @@ -18,198 +18,169 @@ function buildModel(): Model<"openai-responses"> { }; } -function installFailingFetchCapture() { - const originalFetch = globalThis.fetch; - let lastBody: unknown; - - const fetchImpl: typeof fetch = async (_input, init) => { - const rawBody = init?.body; - const bodyText = (() => { - if (!rawBody) { - return ""; - } - if (typeof rawBody === "string") { - return rawBody; - } - if (rawBody instanceof Uint8Array) { - return Buffer.from(rawBody).toString("utf8"); - } - if (rawBody instanceof ArrayBuffer) { - return Buffer.from(new Uint8Array(rawBody)).toString("utf8"); - } - return null; - })(); - lastBody = bodyText ? (JSON.parse(bodyText) as unknown) : undefined; - throw new Error("intentional fetch abort (test)"); - }; - - globalThis.fetch = fetchImpl; - - return { - getLastBody: () => lastBody as Record | undefined, - restore: () => { - globalThis.fetch = originalFetch; - }, - }; -} - describe("openai-responses reasoning replay", () => { it("replays reasoning for tool-call-only turns (OpenAI requires it)", async () => { - const cap = installFailingFetchCapture(); - try { - const model = buildModel(); + const model = buildModel(); + const controller = new AbortController(); + controller.abort(); + let payload: Record | undefined; - const assistantToolOnly: AssistantMessage = { - role: "assistant", - api: "openai-responses", - provider: "openai", - model: "gpt-5.2", - usage: { - input: 0, - output: 0, - cacheRead: 0, - cacheWrite: 0, - totalTokens: 0, - cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 }, + const assistantToolOnly: AssistantMessage = { + role: "assistant", + api: "openai-responses", + provider: "openai", + model: "gpt-5.2", + usage: { + input: 0, + output: 0, + cacheRead: 0, + cacheWrite: 0, + totalTokens: 0, + cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 }, + }, + stopReason: "toolUse", + timestamp: Date.now(), + content: [ + { + type: "thinking", + thinking: "internal", + thinkingSignature: JSON.stringify({ + type: "reasoning", + id: "rs_test", + summary: [], + }), }, - stopReason: "toolUse", - timestamp: Date.now(), - content: [ + { + type: "toolCall", + id: "call_123|fc_123", + name: "noop", + arguments: {}, + }, + ], + }; + + const toolResult: ToolResultMessage = { + role: "toolResult", + toolCallId: "call_123|fc_123", + toolName: "noop", + content: [{ type: "text", text: "ok" }], + isError: false, + timestamp: Date.now(), + }; + + const stream = streamOpenAIResponses( + model, + { + systemPrompt: "system", + messages: [ { - type: "thinking", - thinking: "internal", - thinkingSignature: JSON.stringify({ - type: "reasoning", - id: "rs_test", - summary: [], - }), + role: "user", + content: "Call noop.", + timestamp: Date.now(), }, + assistantToolOnly, + toolResult, { - type: "toolCall", - id: "call_123|fc_123", - name: "noop", - arguments: {}, + role: "user", + content: "Now reply with ok.", + timestamp: Date.now(), }, ], - }; - - const toolResult: ToolResultMessage = { - role: "toolResult", - toolCallId: "call_123|fc_123", - toolName: "noop", - content: [{ type: "text", text: "ok" }], - isError: false, - timestamp: Date.now(), - }; - - const stream = streamOpenAIResponses( - model, - { - systemPrompt: "system", - messages: [ - { - role: "user", - content: "Call noop.", - timestamp: Date.now(), - }, - assistantToolOnly, - toolResult, - { - role: "user", - content: "Now reply with ok.", - timestamp: Date.now(), - }, - ], - tools: [ - { - name: "noop", - description: "no-op", - parameters: Type.Object({}, { additionalProperties: false }), - }, - ], + tools: [ + { + name: "noop", + description: "no-op", + parameters: Type.Object({}, { additionalProperties: false }), + }, + ], + }, + { + apiKey: "test", + signal: controller.signal, + onPayload: (nextPayload) => { + payload = nextPayload as Record; }, - { apiKey: "test" }, - ); + }, + ); - await stream.result(); + await stream.result(); - const body = cap.getLastBody(); - const input = Array.isArray(body?.input) ? body?.input : []; - const types = input - .map((item) => - item && typeof item === "object" ? (item as Record).type : undefined, - ) - .filter((t): t is string => typeof t === "string"); + const input = Array.isArray(payload?.input) ? payload?.input : []; + const types = input + .map((item) => + item && typeof item === "object" ? (item as Record).type : undefined, + ) + .filter((t): t is string => typeof t === "string"); - expect(types).toContain("reasoning"); - expect(types).toContain("function_call"); - expect(types.indexOf("reasoning")).toBeLessThan(types.indexOf("function_call")); - } finally { - cap.restore(); - } + expect(types).toContain("reasoning"); + expect(types).toContain("function_call"); + expect(types.indexOf("reasoning")).toBeLessThan(types.indexOf("function_call")); }); it("still replays reasoning when paired with an assistant message", async () => { - const cap = installFailingFetchCapture(); - try { - const model = buildModel(); + const model = buildModel(); + const controller = new AbortController(); + controller.abort(); + let payload: Record | undefined; - const assistantWithText: AssistantMessage = { - role: "assistant", - api: "openai-responses", - provider: "openai", - model: "gpt-5.2", - usage: { - input: 0, - output: 0, - cacheRead: 0, - cacheWrite: 0, - totalTokens: 0, - cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 }, - }, - stopReason: "stop", - timestamp: Date.now(), - content: [ - { - type: "thinking", - thinking: "internal", - thinkingSignature: JSON.stringify({ - type: "reasoning", - id: "rs_test", - summary: [], - }), - }, - { type: "text", text: "hello", textSignature: "msg_test" }, - ], - }; - - const stream = streamOpenAIResponses( - model, + const assistantWithText: AssistantMessage = { + role: "assistant", + api: "openai-responses", + provider: "openai", + model: "gpt-5.2", + usage: { + input: 0, + output: 0, + cacheRead: 0, + cacheWrite: 0, + totalTokens: 0, + cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 }, + }, + stopReason: "stop", + timestamp: Date.now(), + content: [ { - systemPrompt: "system", - messages: [ - { role: "user", content: "Hi", timestamp: Date.now() }, - assistantWithText, - { role: "user", content: "Ok", timestamp: Date.now() }, - ], + type: "thinking", + thinking: "internal", + thinkingSignature: JSON.stringify({ + type: "reasoning", + id: "rs_test", + summary: [], + }), }, - { apiKey: "test" }, - ); + { type: "text", text: "hello", textSignature: "msg_test" }, + ], + }; - await stream.result(); + const stream = streamOpenAIResponses( + model, + { + systemPrompt: "system", + messages: [ + { role: "user", content: "Hi", timestamp: Date.now() }, + assistantWithText, + { role: "user", content: "Ok", timestamp: Date.now() }, + ], + }, + { + apiKey: "test", + signal: controller.signal, + onPayload: (nextPayload) => { + payload = nextPayload as Record; + }, + }, + ); - const body = cap.getLastBody(); - const input = Array.isArray(body?.input) ? body?.input : []; - const types = input - .map((item) => - item && typeof item === "object" ? (item as Record).type : undefined, - ) - .filter((t): t is string => typeof t === "string"); + await stream.result(); - expect(types).toContain("reasoning"); - expect(types).toContain("message"); - } finally { - cap.restore(); - } + const input = Array.isArray(payload?.input) ? payload?.input : []; + const types = input + .map((item) => + item && typeof item === "object" ? (item as Record).type : undefined, + ) + .filter((t): t is string => typeof t === "string"); + + expect(types).toContain("reasoning"); + expect(types).toContain("message"); }); }); diff --git a/src/agents/pi-embedded-runner.get-dm-history-limit-from-session-key.returns-undefined-sessionkey-is-undefined.e2e.test.ts b/src/agents/pi-embedded-runner.get-dm-history-limit-from-session-key.returns-undefined-sessionkey-is-undefined.e2e.test.ts index 15aece8c26e..b685f88d499 100644 --- a/src/agents/pi-embedded-runner.get-dm-history-limit-from-session-key.returns-undefined-sessionkey-is-undefined.e2e.test.ts +++ b/src/agents/pi-embedded-runner.get-dm-history-limit-from-session-key.returns-undefined-sessionkey-is-undefined.e2e.test.ts @@ -143,14 +143,23 @@ describe("getDmHistoryLimitFromSessionKey", () => { 9, ); }); - it("returns undefined for non-dm session kinds", () => { + it("returns historyLimit for channel session kinds when configured", () => { const config = { channels: { - telegram: { dmHistoryLimit: 15 }, - slack: { dmHistoryLimit: 10 }, + slack: { historyLimit: 10, dmHistoryLimit: 15 }, + discord: { historyLimit: 8 }, }, } as OpenClawConfig; - expect(getDmHistoryLimitFromSessionKey("agent:beta:slack:channel:c1", config)).toBeUndefined(); + expect(getDmHistoryLimitFromSessionKey("agent:beta:slack:channel:c1", config)).toBe(10); + expect(getDmHistoryLimitFromSessionKey("discord:channel:123456", config)).toBe(8); + }); + it("returns undefined for non-dm/channel/group session kinds", () => { + const config = { + channels: { + telegram: { dmHistoryLimit: 15, historyLimit: 10 }, + }, + } as OpenClawConfig; + // "slash" is not dm, channel, or group expect(getDmHistoryLimitFromSessionKey("telegram:slash:123", config)).toBeUndefined(); }); it("returns undefined for unknown provider", () => { @@ -228,6 +237,46 @@ describe("getDmHistoryLimitFromSessionKey", () => { } as OpenClawConfig; expect(getDmHistoryLimitFromSessionKey("telegram:dm:123", config)).toBe(5); }); + it("returns historyLimit for channel sessions for all providers", () => { + const providers = [ + "telegram", + "whatsapp", + "discord", + "slack", + "signal", + "imessage", + "msteams", + "nextcloud-talk", + ] as const; + + for (const provider of providers) { + const config = { + channels: { [provider]: { historyLimit: 12 } }, + } as OpenClawConfig; + expect(getDmHistoryLimitFromSessionKey(`${provider}:channel:123`, config)).toBe(12); + expect(getDmHistoryLimitFromSessionKey(`agent:main:${provider}:channel:456`, config)).toBe( + 12, + ); + } + }); + it("returns historyLimit for group sessions", () => { + const config = { + channels: { + discord: { historyLimit: 15 }, + slack: { historyLimit: 10 }, + }, + } as OpenClawConfig; + expect(getDmHistoryLimitFromSessionKey("discord:group:123", config)).toBe(15); + expect(getDmHistoryLimitFromSessionKey("agent:main:slack:group:abc", config)).toBe(10); + }); + it("returns undefined for channel sessions when historyLimit is not configured", () => { + const config = { + channels: { + discord: { dmHistoryLimit: 10 }, // only dmHistoryLimit, no historyLimit + }, + } as OpenClawConfig; + expect(getDmHistoryLimitFromSessionKey("discord:channel:123", config)).toBeUndefined(); + }); describe("backward compatibility", () => { it("accepts both legacy :dm: and new :direct: session keys", () => { diff --git a/src/agents/pi-embedded-runner.history-limit-from-session-key.test.ts b/src/agents/pi-embedded-runner.history-limit-from-session-key.test.ts new file mode 100644 index 00000000000..776c54f1c6e --- /dev/null +++ b/src/agents/pi-embedded-runner.history-limit-from-session-key.test.ts @@ -0,0 +1,31 @@ +import { describe, expect, it } from "vitest"; +import type { OpenClawConfig } from "../config/config.js"; +import { getDmHistoryLimitFromSessionKey } from "./pi-embedded-runner.js"; + +describe("getDmHistoryLimitFromSessionKey", () => { + it("keeps backward compatibility for dm/direct session kinds", () => { + const config = { + channels: { telegram: { dmHistoryLimit: 10 } }, + } as OpenClawConfig; + + expect(getDmHistoryLimitFromSessionKey("telegram:dm:123", config)).toBe(10); + expect(getDmHistoryLimitFromSessionKey("telegram:direct:123", config)).toBe(10); + }); + + it("returns historyLimit for channel and group session kinds", () => { + const config = { + channels: { discord: { historyLimit: 12, dmHistoryLimit: 5 } }, + } as OpenClawConfig; + + expect(getDmHistoryLimitFromSessionKey("discord:channel:123", config)).toBe(12); + expect(getDmHistoryLimitFromSessionKey("discord:group:456", config)).toBe(12); + }); + + it("returns undefined for unsupported session kinds", () => { + const config = { + channels: { discord: { historyLimit: 12, dmHistoryLimit: 5 } }, + } as OpenClawConfig; + + expect(getDmHistoryLimitFromSessionKey("discord:slash:123", config)).toBeUndefined(); + }); +}); diff --git a/src/agents/pi-embedded-runner.ts b/src/agents/pi-embedded-runner.ts index bdebd000522..4d968a9c2eb 100644 --- a/src/agents/pi-embedded-runner.ts +++ b/src/agents/pi-embedded-runner.ts @@ -5,6 +5,7 @@ export { applyExtraParamsToAgent, resolveExtraParams } from "./pi-embedded-runne export { applyGoogleTurnOrderingFix } from "./pi-embedded-runner/google.js"; export { getDmHistoryLimitFromSessionKey, + getHistoryLimitFromSessionKey, limitHistoryTurns, } from "./pi-embedded-runner/history.js"; export { resolveEmbeddedSessionLane } from "./pi-embedded-runner/lanes.js"; diff --git a/src/agents/pi-embedded-runner/compact.ts b/src/agents/pi-embedded-runner/compact.ts index 0eec28249ce..1ada5c626a5 100644 --- a/src/agents/pi-embedded-runner/compact.ts +++ b/src/agents/pi-embedded-runner/compact.ts @@ -1,3 +1,4 @@ +import type { AgentMessage } from "@mariozechner/pi-agent-core"; import { createAgentSession, estimateTokens, @@ -13,6 +14,7 @@ import type { EmbeddedPiCompactResult } from "./types.js"; import { resolveHeartbeatPrompt } from "../../auto-reply/heartbeat.js"; import { resolveChannelCapabilities } from "../../config/channel-capabilities.js"; import { getMachineDisplayName } from "../../infra/machine-name.js"; +import { getGlobalHookRunner } from "../../plugins/hook-runner-global.js"; import { type enqueueCommand, enqueueCommandInLane } from "../../process/command-queue.js"; import { isSubagentSessionKey } from "../../routing/session-key.js"; import { resolveSignalReactionLevel } from "../../signal/reaction-level.js"; @@ -73,11 +75,12 @@ import { createSystemPromptOverride, } from "./system-prompt.js"; import { splitSdkTools } from "./tool-split.js"; -import { describeUnknownError, mapThinkingLevel, resolveExecToolDefaults } from "./utils.js"; +import { describeUnknownError, mapThinkingLevel } from "./utils.js"; import { flushPendingToolResultsAfterIdle } from "./wait-for-idle-before-flush.js"; export type CompactEmbeddedPiSessionParams = { sessionId: string; + runId?: string; sessionKey?: string; messageChannel?: string; messageProvider?: string; @@ -104,12 +107,132 @@ export type CompactEmbeddedPiSessionParams = { reasoningLevel?: ReasoningLevel; bashElevated?: ExecElevatedDefaults; customInstructions?: string; + trigger?: "overflow" | "manual" | "cache_ttl" | "safeguard"; + diagId?: string; + attempt?: number; + maxAttempts?: number; lane?: string; enqueue?: typeof enqueueCommand; extraSystemPrompt?: string; ownerNumbers?: string[]; }; +type CompactionMessageMetrics = { + messages: number; + historyTextChars: number; + toolResultChars: number; + estTokens?: number; + contributors: Array<{ role: string; chars: number; tool?: string }>; +}; + +function createCompactionDiagId(): string { + return `cmp-${Date.now().toString(36)}-${Math.random().toString(36).slice(2, 8)}`; +} + +function getMessageTextChars(msg: AgentMessage): number { + const content = (msg as { content?: unknown }).content; + if (typeof content === "string") { + return content.length; + } + if (!Array.isArray(content)) { + return 0; + } + let total = 0; + for (const block of content) { + if (!block || typeof block !== "object") { + continue; + } + const text = (block as { text?: unknown }).text; + if (typeof text === "string") { + total += text.length; + } + } + return total; +} + +function resolveMessageToolLabel(msg: AgentMessage): string | undefined { + const candidate = + (msg as { toolName?: unknown }).toolName ?? + (msg as { name?: unknown }).name ?? + (msg as { tool?: unknown }).tool; + return typeof candidate === "string" && candidate.trim().length > 0 ? candidate : undefined; +} + +function summarizeCompactionMessages(messages: AgentMessage[]): CompactionMessageMetrics { + let historyTextChars = 0; + let toolResultChars = 0; + const contributors: Array<{ role: string; chars: number; tool?: string }> = []; + let estTokens = 0; + let tokenEstimationFailed = false; + + for (const msg of messages) { + const role = typeof msg.role === "string" ? msg.role : "unknown"; + const chars = getMessageTextChars(msg); + historyTextChars += chars; + if (role === "toolResult") { + toolResultChars += chars; + } + contributors.push({ role, chars, tool: resolveMessageToolLabel(msg) }); + if (!tokenEstimationFailed) { + try { + estTokens += estimateTokens(msg); + } catch { + tokenEstimationFailed = true; + } + } + } + + return { + messages: messages.length, + historyTextChars, + toolResultChars, + estTokens: tokenEstimationFailed ? undefined : estTokens, + contributors: contributors.toSorted((a, b) => b.chars - a.chars).slice(0, 3), + }; +} + +function classifyCompactionReason(reason?: string): string { + const text = (reason ?? "").trim().toLowerCase(); + if (!text) { + return "unknown"; + } + if (text.includes("nothing to compact")) { + return "no_compactable_entries"; + } + if (text.includes("below threshold")) { + return "below_threshold"; + } + if (text.includes("already compacted")) { + return "already_compacted_recently"; + } + if (text.includes("guard")) { + return "guard_blocked"; + } + if (text.includes("summary")) { + return "summary_failed"; + } + if (text.includes("timed out") || text.includes("timeout")) { + return "timeout"; + } + if ( + text.includes("400") || + text.includes("401") || + text.includes("403") || + text.includes("429") + ) { + return "provider_error_4xx"; + } + if ( + text.includes("500") || + text.includes("502") || + text.includes("503") || + text.includes("504") + ) { + return "provider_error_5xx"; + } + return "unknown"; +} + /** * Core compaction logic without lane queueing. * Use this when already inside a session/global lane to avoid deadlocks. @@ -117,6 +240,12 @@ export type CompactEmbeddedPiSessionParams = { export async function compactEmbeddedPiSessionDirect( params: CompactEmbeddedPiSessionParams, ): Promise { + const startedAt = Date.now(); + const diagId = params.diagId?.trim() || createCompactionDiagId(); + const trigger = params.trigger ?? "manual"; + const attempt = params.attempt ?? 1; + const maxAttempts = params.maxAttempts ?? 1; + const runId = params.runId ?? params.sessionId; const resolvedWorkspace = resolveUserPath(params.workspaceDir); const prevCwd = process.cwd(); @@ -131,10 +260,17 @@ export async function compactEmbeddedPiSessionDirect( params.config, ); if (!model) { + const reason = error ?? `Unknown model: ${provider}/${modelId}`; + log.warn( + `[compaction-diag] end runId=${runId} sessionKey=${params.sessionKey ?? params.sessionId} ` + + `diagId=${diagId} trigger=${trigger} provider=${provider}/${modelId} ` + + `attempt=${attempt} maxAttempts=${maxAttempts} outcome=failed reason=${classifyCompactionReason(reason)} ` + + `durationMs=${Date.now() - startedAt}`, + ); return { ok: false, compacted: false, - reason: error ?? `Unknown model: ${provider}/${modelId}`, + reason, }; } try { @@ -161,10 +297,17 @@ export async function compactEmbeddedPiSessionDirect( authStorage.setRuntimeApiKey(model.provider, apiKeyInfo.apiKey); } } catch (err) { + const reason = describeUnknownError(err); + log.warn( + `[compaction-diag] end runId=${runId} sessionKey=${params.sessionKey ?? params.sessionId} ` + + `diagId=${diagId} trigger=${trigger} provider=${provider}/${modelId} ` + + `attempt=${attempt} maxAttempts=${maxAttempts} outcome=failed reason=${classifyCompactionReason(reason)} ` + + `durationMs=${Date.now() - startedAt}`, + ); return { ok: false, compacted: false, - reason: describeUnknownError(err), + reason, }; } @@ -221,7 +364,6 @@ export async function compactEmbeddedPiSessionDirect( const runAbortController = new AbortController(); const toolsRaw = createOpenClawCodingTools({ exec: { - ...resolveExecToolDefaults(params.config), elevated: params.bashElevated, }, sandbox, @@ -431,6 +573,8 @@ export async function compactEmbeddedPiSessionDirect( const validated = transcriptPolicy.validateAnthropicTurns ? validateAnthropicTurns(validatedGemini) : validatedGemini; + // Capture full message history BEFORE limiting β€” plugins need the complete conversation + const preCompactionMessages = [...session.messages]; const truncated = limitHistoryTurns( validated, getDmHistoryLimitFromSessionKey(params.sessionKey, params.config), @@ -444,6 +588,50 @@ export async function compactEmbeddedPiSessionDirect( if (limited.length > 0) { session.agent.replaceMessages(limited); } + // Run before_compaction hooks (fire-and-forget). + // The session JSONL already contains all messages on disk, so plugins + // can read sessionFile asynchronously and process in parallel with + // the compaction LLM call β€” no need to block or wait for after_compaction. + const hookRunner = getGlobalHookRunner(); + const hookCtx = { + agentId: params.sessionKey?.split(":")[0] ?? "main", + sessionKey: params.sessionKey, + sessionId: params.sessionId, + workspaceDir: params.workspaceDir, + messageProvider: params.messageChannel ?? params.messageProvider, + }; + if (hookRunner?.hasHooks("before_compaction")) { + hookRunner + .runBeforeCompaction( + { + messageCount: preCompactionMessages.length, + compactingCount: limited.length, + messages: preCompactionMessages, + sessionFile: params.sessionFile, + }, + hookCtx, + ) + .catch((hookErr: unknown) => { + log.warn(`before_compaction hook failed: ${String(hookErr)}`); + }); + } + + const diagEnabled = log.isEnabled("debug"); + const preMetrics = diagEnabled ? summarizeCompactionMessages(session.messages) : undefined; + if (diagEnabled && preMetrics) { + log.debug( + `[compaction-diag] start runId=${runId} sessionKey=${params.sessionKey ?? params.sessionId} ` + + `diagId=${diagId} trigger=${trigger} provider=${provider}/${modelId} ` + + `attempt=${attempt} maxAttempts=${maxAttempts} ` + + `pre.messages=${preMetrics.messages} pre.historyTextChars=${preMetrics.historyTextChars} ` + + `pre.toolResultChars=${preMetrics.toolResultChars} pre.estTokens=${preMetrics.estTokens ?? "unknown"}`, + ); + log.debug( + `[compaction-diag] contributors diagId=${diagId} top=${JSON.stringify(preMetrics.contributors)}`, + ); + } + + const compactStartedAt = Date.now(); const result = await session.compact(params.customInstructions); // Estimate tokens after compaction by summing token estimates for remaining messages let tokensAfter: number | undefined; @@ -460,6 +648,40 @@ export async function compactEmbeddedPiSessionDirect( // If estimation fails, leave tokensAfter undefined tokensAfter = undefined; } + // Run after_compaction hooks (fire-and-forget). + // Also includes sessionFile for plugins that only need to act after + // compaction completes (e.g. analytics, cleanup). + if (hookRunner?.hasHooks("after_compaction")) { + hookRunner + .runAfterCompaction( + { + messageCount: session.messages.length, + tokenCount: tokensAfter, + compactedCount: limited.length - session.messages.length, + sessionFile: params.sessionFile, + }, + hookCtx, + ) + .catch((hookErr) => { + log.warn(`after_compaction hook failed: ${hookErr}`); + }); + } + + const postMetrics = diagEnabled ? summarizeCompactionMessages(session.messages) : undefined; + if (diagEnabled && preMetrics && postMetrics) { + log.debug( + `[compaction-diag] end runId=${runId} sessionKey=${params.sessionKey ?? params.sessionId} ` + + `diagId=${diagId} trigger=${trigger} provider=${provider}/${modelId} ` + + `attempt=${attempt} maxAttempts=${maxAttempts} outcome=compacted reason=none ` + + `durationMs=${Date.now() - compactStartedAt} retrying=false ` + + `post.messages=${postMetrics.messages} post.historyTextChars=${postMetrics.historyTextChars} ` + + `post.toolResultChars=${postMetrics.toolResultChars} post.estTokens=${postMetrics.estTokens ?? "unknown"} ` + + `delta.messages=${postMetrics.messages - preMetrics.messages} ` + + `delta.historyTextChars=${postMetrics.historyTextChars - preMetrics.historyTextChars} ` + + `delta.toolResultChars=${postMetrics.toolResultChars - preMetrics.toolResultChars} ` + + `delta.estTokens=${typeof preMetrics.estTokens === "number" && typeof postMetrics.estTokens === "number" ? postMetrics.estTokens - preMetrics.estTokens : "unknown"}`, + ); + } return { ok: true, compacted: true, @@ -482,10 +704,17 @@ export async function compactEmbeddedPiSessionDirect( await sessionLock.release(); } } catch (err) { + const reason = describeUnknownError(err); + log.warn( + `[compaction-diag] end runId=${runId} sessionKey=${params.sessionKey ?? params.sessionId} ` + + `diagId=${diagId} trigger=${trigger} provider=${provider}/${modelId} ` + + `attempt=${attempt} maxAttempts=${maxAttempts} outcome=failed reason=${classifyCompactionReason(reason)} ` + + `durationMs=${Date.now() - startedAt}`, + ); return { ok: false, compacted: false, - reason: describeUnknownError(err), + reason, }; } finally { restoreSkillEnv?.(); diff --git a/src/agents/pi-embedded-runner/history.ts b/src/agents/pi-embedded-runner/history.ts index 0340c315cc7..6515c0c13d5 100644 --- a/src/agents/pi-embedded-runner/history.ts +++ b/src/agents/pi-embedded-runner/history.ts @@ -38,8 +38,9 @@ export function limitHistoryTurns( /** * Extract provider + user ID from a session key and look up dmHistoryLimit. * Supports per-DM overrides and provider defaults. + * For channel/group sessions, uses historyLimit from provider config. */ -export function getDmHistoryLimitFromSessionKey( +export function getHistoryLimitFromSessionKey( sessionKey: string | undefined, config: OpenClawConfig | undefined, ): number | undefined { @@ -58,32 +59,17 @@ export function getDmHistoryLimitFromSessionKey( const kind = providerParts[1]?.toLowerCase(); const userIdRaw = providerParts.slice(2).join(":"); const userId = stripThreadSuffix(userIdRaw); - // Accept both "direct" (new) and "dm" (legacy) for backward compat - if (kind !== "direct" && kind !== "dm") { - return undefined; - } - - const getLimit = ( - providerConfig: - | { - dmHistoryLimit?: number; - dms?: Record; - } - | undefined, - ): number | undefined => { - if (!providerConfig) { - return undefined; - } - if (userId && providerConfig.dms?.[userId]?.historyLimit !== undefined) { - return providerConfig.dms[userId].historyLimit; - } - return providerConfig.dmHistoryLimit; - }; const resolveProviderConfig = ( cfg: OpenClawConfig | undefined, providerId: string, - ): { dmHistoryLimit?: number; dms?: Record } | undefined => { + ): + | { + historyLimit?: number; + dmHistoryLimit?: number; + dms?: Record; + } + | undefined => { const channels = cfg?.channels; if (!channels || typeof channels !== "object") { return undefined; @@ -92,8 +78,38 @@ export function getDmHistoryLimitFromSessionKey( if (!entry || typeof entry !== "object" || Array.isArray(entry)) { return undefined; } - return entry as { dmHistoryLimit?: number; dms?: Record }; + return entry as { + historyLimit?: number; + dmHistoryLimit?: number; + dms?: Record; + }; }; - return getLimit(resolveProviderConfig(config, provider)); + const providerConfig = resolveProviderConfig(config, provider); + if (!providerConfig) { + return undefined; + } + + // For DM sessions: per-DM override -> dmHistoryLimit. + // Accept both "direct" (new) and "dm" (legacy) for backward compat. + if (kind === "dm" || kind === "direct") { + if (userId && providerConfig.dms?.[userId]?.historyLimit !== undefined) { + return providerConfig.dms[userId].historyLimit; + } + return providerConfig.dmHistoryLimit; + } + + // For channel/group sessions: use historyLimit from provider config + // This prevents context overflow in long-running channel sessions + if (kind === "channel" || kind === "group") { + return providerConfig.historyLimit; + } + + return undefined; } + +/** + * @deprecated Use getHistoryLimitFromSessionKey instead. + * Alias for backward compatibility. + */ +export const getDmHistoryLimitFromSessionKey = getHistoryLimitFromSessionKey; diff --git a/src/agents/pi-embedded-runner/model.test.ts b/src/agents/pi-embedded-runner/model.test.ts index 69c93ca8cfd..794b4c3d985 100644 --- a/src/agents/pi-embedded-runner/model.test.ts +++ b/src/agents/pi-embedded-runner/model.test.ts @@ -172,43 +172,6 @@ describe("resolveModel", () => { }); }); - it("builds an openai-codex fallback for gpt-5.3-codex-spark", () => { - const templateModel = { - id: "gpt-5.2-codex", - name: "GPT-5.2 Codex", - provider: "openai-codex", - api: "openai-codex-responses", - baseUrl: "https://chatgpt.com/backend-api", - reasoning: true, - input: ["text", "image"] as const, - cost: { input: 1.75, output: 14, cacheRead: 0.175, cacheWrite: 0 }, - contextWindow: 272000, - maxTokens: 128000, - }; - - vi.mocked(discoverModels).mockReturnValue({ - find: vi.fn((provider: string, modelId: string) => { - if (provider === "openai-codex" && modelId === "gpt-5.2-codex") { - return templateModel; - } - return null; - }), - } as unknown as ReturnType); - - const result = resolveModel("openai-codex", "gpt-5.3-codex-spark", "/tmp/agent"); - - expect(result.error).toBeUndefined(); - expect(result.model).toMatchObject({ - provider: "openai-codex", - id: "gpt-5.3-codex-spark", - api: "openai-codex-responses", - baseUrl: "https://chatgpt.com/backend-api", - reasoning: true, - contextWindow: 272000, - maxTokens: 128000, - }); - }); - it("builds an anthropic forward-compat fallback for claude-opus-4-6", () => { const templateModel = { id: "claude-opus-4-5", @@ -244,7 +207,7 @@ describe("resolveModel", () => { }); }); - it("builds a google-antigravity forward-compat fallback for claude-opus-4-6-thinking", () => { + it("builds an antigravity forward-compat fallback for claude-opus-4-6-thinking", () => { const templateModel = { id: "claude-opus-4-5-thinking", name: "Claude Opus 4.5 Thinking", @@ -253,8 +216,8 @@ describe("resolveModel", () => { baseUrl: "https://daily-cloudcode-pa.sandbox.googleapis.com", reasoning: true, input: ["text", "image"] as const, - cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 }, - contextWindow: 1000000, + cost: { input: 5, output: 25, cacheRead: 0.5, cacheWrite: 6.25 }, + contextWindow: 200000, maxTokens: 64000, }; @@ -276,6 +239,45 @@ describe("resolveModel", () => { api: "google-gemini-cli", baseUrl: "https://daily-cloudcode-pa.sandbox.googleapis.com", reasoning: true, + contextWindow: 200000, + maxTokens: 64000, + }); + }); + + it("builds an antigravity forward-compat fallback for claude-opus-4-6", () => { + const templateModel = { + id: "claude-opus-4-5", + name: "Claude Opus 4.5", + provider: "google-antigravity", + api: "google-gemini-cli", + baseUrl: "https://daily-cloudcode-pa.sandbox.googleapis.com", + reasoning: true, + input: ["text", "image"] as const, + cost: { input: 5, output: 25, cacheRead: 0.5, cacheWrite: 6.25 }, + contextWindow: 200000, + maxTokens: 64000, + }; + + vi.mocked(discoverModels).mockReturnValue({ + find: vi.fn((provider: string, modelId: string) => { + if (provider === "google-antigravity" && modelId === "claude-opus-4-5") { + return templateModel; + } + return null; + }), + } as unknown as ReturnType); + + const result = resolveModel("google-antigravity", "claude-opus-4-6", "/tmp/agent"); + + expect(result.error).toBeUndefined(); + expect(result.model).toMatchObject({ + provider: "google-antigravity", + id: "claude-opus-4-6", + api: "google-gemini-cli", + baseUrl: "https://daily-cloudcode-pa.sandbox.googleapis.com", + reasoning: true, + contextWindow: 200000, + maxTokens: 64000, }); }); @@ -314,18 +316,34 @@ describe("resolveModel", () => { }); }); + it("keeps unknown-model errors when no antigravity thinking template exists", () => { + vi.mocked(discoverModels).mockReturnValue({ + find: vi.fn(() => null), + } as unknown as ReturnType); + + const result = resolveModel("google-antigravity", "claude-opus-4-6-thinking", "/tmp/agent"); + + expect(result.model).toBeUndefined(); + expect(result.error).toBe("Unknown model: google-antigravity/claude-opus-4-6-thinking"); + }); + + it("keeps unknown-model errors when no antigravity non-thinking template exists", () => { + vi.mocked(discoverModels).mockReturnValue({ + find: vi.fn(() => null), + } as unknown as ReturnType); + + const result = resolveModel("google-antigravity", "claude-opus-4-6", "/tmp/agent"); + + expect(result.model).toBeUndefined(); + expect(result.error).toBe("Unknown model: google-antigravity/claude-opus-4-6"); + }); + it("keeps unknown-model errors for non-gpt-5 openai-codex ids", () => { const result = resolveModel("openai-codex", "gpt-4.1-mini", "/tmp/agent"); expect(result.model).toBeUndefined(); expect(result.error).toBe("Unknown model: openai-codex/gpt-4.1-mini"); }); - it("errors for unknown gpt-5.3-codex-* variants", () => { - const result = resolveModel("openai-codex", "gpt-5.3-codex-unknown", "/tmp/agent"); - expect(result.model).toBeUndefined(); - expect(result.error).toBe("Unknown model: openai-codex/gpt-5.3-codex-unknown"); - }); - it("uses codex fallback even when openai-codex provider is configured", () => { // This test verifies the ordering: codex fallback must fire BEFORE the generic providerCfg fallback. // If ordering is wrong, the generic fallback would use api: "openai-responses" (the default) diff --git a/src/agents/pi-embedded-runner/model.ts b/src/agents/pi-embedded-runner/model.ts index 41e1f8baf10..cbc21fe2d4f 100644 --- a/src/agents/pi-embedded-runner/model.ts +++ b/src/agents/pi-embedded-runner/model.ts @@ -4,6 +4,7 @@ import type { ModelDefinitionConfig } from "../../config/types.js"; import { resolveOpenClawAgentDir } from "../agent-paths.js"; import { DEFAULT_CONTEXT_TOKENS } from "../defaults.js"; import { normalizeModelCompat } from "../model-compat.js"; +import { resolveForwardCompatModel } from "../model-forward-compat.js"; import { normalizeProviderId } from "../model-selection.js"; import { discoverAuthStorage, @@ -19,188 +20,6 @@ type InlineProviderConfig = { models?: ModelDefinitionConfig[]; }; -const OPENAI_CODEX_GPT_53_MODEL_ID = "gpt-5.3-codex"; -const OPENAI_CODEX_GPT_53_SPARK_MODEL_ID = "gpt-5.3-codex-spark"; - -const OPENAI_CODEX_TEMPLATE_MODEL_IDS = ["gpt-5.2-codex"] as const; - -// pi-ai's built-in Anthropic catalog can lag behind OpenClaw's defaults/docs. -// Add forward-compat fallbacks for known-new IDs by cloning an older template model. -const ANTHROPIC_OPUS_46_MODEL_ID = "claude-opus-4-6"; -const ANTHROPIC_OPUS_46_DOT_MODEL_ID = "claude-opus-4.6"; -const ANTHROPIC_OPUS_TEMPLATE_MODEL_IDS = ["claude-opus-4-5", "claude-opus-4.5"] as const; - -function resolveOpenAICodexGpt53FallbackModel( - provider: string, - modelId: string, - modelRegistry: ModelRegistry, -): Model | undefined { - const normalizedProvider = normalizeProviderId(provider); - const trimmedModelId = modelId.trim(); - if (normalizedProvider !== "openai-codex") { - return undefined; - } - - const lower = trimmedModelId.toLowerCase(); - const isGpt53 = lower === OPENAI_CODEX_GPT_53_MODEL_ID; - const isSpark = lower === OPENAI_CODEX_GPT_53_SPARK_MODEL_ID; - if (!isGpt53 && !isSpark) { - return undefined; - } - - for (const templateId of OPENAI_CODEX_TEMPLATE_MODEL_IDS) { - const template = modelRegistry.find(normalizedProvider, templateId) as Model | null; - if (!template) { - continue; - } - return normalizeModelCompat({ - ...template, - id: trimmedModelId, - name: trimmedModelId, - // Spark is a low-latency variant; keep api/baseUrl from template. - ...(isSpark ? { reasoning: true } : {}), - } as Model); - } - - return normalizeModelCompat({ - id: trimmedModelId, - name: trimmedModelId, - api: "openai-codex-responses", - provider: normalizedProvider, - baseUrl: "https://chatgpt.com/backend-api", - reasoning: true, - input: ["text", "image"], - cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 }, - contextWindow: DEFAULT_CONTEXT_TOKENS, - maxTokens: DEFAULT_CONTEXT_TOKENS, - } as Model); -} - -function resolveAnthropicOpus46ForwardCompatModel( - provider: string, - modelId: string, - modelRegistry: ModelRegistry, -): Model | undefined { - const normalizedProvider = normalizeProviderId(provider); - if (normalizedProvider !== "anthropic") { - return undefined; - } - - const trimmedModelId = modelId.trim(); - const lower = trimmedModelId.toLowerCase(); - const isOpus46 = - lower === ANTHROPIC_OPUS_46_MODEL_ID || - lower === ANTHROPIC_OPUS_46_DOT_MODEL_ID || - lower.startsWith(`${ANTHROPIC_OPUS_46_MODEL_ID}-`) || - lower.startsWith(`${ANTHROPIC_OPUS_46_DOT_MODEL_ID}-`); - if (!isOpus46) { - return undefined; - } - - const templateIds: string[] = []; - if (lower.startsWith(ANTHROPIC_OPUS_46_MODEL_ID)) { - templateIds.push(lower.replace(ANTHROPIC_OPUS_46_MODEL_ID, "claude-opus-4-5")); - } - if (lower.startsWith(ANTHROPIC_OPUS_46_DOT_MODEL_ID)) { - templateIds.push(lower.replace(ANTHROPIC_OPUS_46_DOT_MODEL_ID, "claude-opus-4.5")); - } - templateIds.push(...ANTHROPIC_OPUS_TEMPLATE_MODEL_IDS); - - for (const templateId of [...new Set(templateIds)].filter(Boolean)) { - const template = modelRegistry.find(normalizedProvider, templateId) as Model | null; - if (!template) { - continue; - } - return normalizeModelCompat({ - ...template, - id: trimmedModelId, - name: trimmedModelId, - } as Model); - } - - return undefined; -} - -// Z.ai's GLM-5 may not be present in pi-ai's built-in model catalog yet. -// When a user configures zai/glm-5 without a models.json entry, clone glm-4.7 as a forward-compat fallback. -const ZAI_GLM5_MODEL_ID = "glm-5"; -const ZAI_GLM5_TEMPLATE_MODEL_IDS = ["glm-4.7"] as const; - -function resolveZaiGlm5ForwardCompatModel( - provider: string, - modelId: string, - modelRegistry: ModelRegistry, -): Model | undefined { - if (normalizeProviderId(provider) !== "zai") { - return undefined; - } - const trimmed = modelId.trim(); - const lower = trimmed.toLowerCase(); - if (lower !== ZAI_GLM5_MODEL_ID && !lower.startsWith(`${ZAI_GLM5_MODEL_ID}-`)) { - return undefined; - } - - for (const templateId of ZAI_GLM5_TEMPLATE_MODEL_IDS) { - const template = modelRegistry.find("zai", templateId) as Model | null; - if (!template) { - continue; - } - return normalizeModelCompat({ - ...template, - id: trimmed, - name: trimmed, - reasoning: true, - } as Model); - } - - return normalizeModelCompat({ - id: trimmed, - name: trimmed, - api: "openai-completions", - provider: "zai", - reasoning: true, - input: ["text"], - cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 }, - contextWindow: DEFAULT_CONTEXT_TOKENS, - maxTokens: DEFAULT_CONTEXT_TOKENS, - } as Model); -} - -// google-antigravity's model catalog in pi-ai can lag behind the actual platform. -// When a google-antigravity model ID contains "opus-4-6" (or "opus-4.6") but isn't -// in the registry yet, clone the opus-4-5 template so the correct api -// ("google-gemini-cli") and baseUrl are preserved. -const ANTIGRAVITY_OPUS_46_STEMS = ["claude-opus-4-6", "claude-opus-4.6"] as const; -const ANTIGRAVITY_OPUS_45_TEMPLATES = ["claude-opus-4-5-thinking", "claude-opus-4-5"] as const; - -function resolveAntigravityOpus46ForwardCompatModel( - provider: string, - modelId: string, - modelRegistry: ModelRegistry, -): Model | undefined { - if (normalizeProviderId(provider) !== "google-antigravity") { - return undefined; - } - const lower = modelId.trim().toLowerCase(); - const isOpus46 = ANTIGRAVITY_OPUS_46_STEMS.some( - (stem) => lower === stem || lower.startsWith(`${stem}-`), - ); - if (!isOpus46) { - return undefined; - } - for (const templateId of ANTIGRAVITY_OPUS_45_TEMPLATES) { - const template = modelRegistry.find("google-antigravity", templateId) as Model | null; - if (template) { - return normalizeModelCompat({ - ...template, - id: modelId.trim(), - name: modelId.trim(), - } as Model); - } - } - return undefined; -} - export function buildInlineProviderModels( providers: Record, ): InlineModelEntry[] { @@ -267,36 +86,11 @@ export function resolveModel( modelRegistry, }; } - // Codex gpt-5.3 forward-compat fallback must be checked BEFORE the generic providerCfg fallback. - // Otherwise, if cfg.models.providers["openai-codex"] is configured, the generic fallback fires - // with api: "openai-responses" instead of the correct "openai-codex-responses". - const codexForwardCompat = resolveOpenAICodexGpt53FallbackModel( - provider, - modelId, - modelRegistry, - ); - if (codexForwardCompat) { - return { model: codexForwardCompat, authStorage, modelRegistry }; - } - const anthropicForwardCompat = resolveAnthropicOpus46ForwardCompatModel( - provider, - modelId, - modelRegistry, - ); - if (anthropicForwardCompat) { - return { model: anthropicForwardCompat, authStorage, modelRegistry }; - } - const antigravityForwardCompat = resolveAntigravityOpus46ForwardCompatModel( - provider, - modelId, - modelRegistry, - ); - if (antigravityForwardCompat) { - return { model: antigravityForwardCompat, authStorage, modelRegistry }; - } - const zaiForwardCompat = resolveZaiGlm5ForwardCompatModel(provider, modelId, modelRegistry); - if (zaiForwardCompat) { - return { model: zaiForwardCompat, authStorage, modelRegistry }; + // Forward-compat fallbacks must be checked BEFORE the generic providerCfg fallback. + // Otherwise, configured providers can default to a generic API and break specific transports. + const forwardCompat = resolveForwardCompatModel(provider, modelId, modelRegistry); + if (forwardCompat) { + return { model: forwardCompat, authStorage, modelRegistry }; } const providerCfg = providers[provider]; if (providerCfg || modelId.startsWith("mock-")) { diff --git a/src/agents/pi-embedded-runner/run.ts b/src/agents/pi-embedded-runner/run.ts index 6cbd3dd4cab..96dc8db0379 100644 --- a/src/agents/pi-embedded-runner/run.ts +++ b/src/agents/pi-embedded-runner/run.ts @@ -97,6 +97,10 @@ const createUsageAccumulator = (): UsageAccumulator => ({ lastInput: 0, }); +function createCompactionDiagId(): string { + return `ovf-${Date.now().toString(36)}-${Math.random().toString(36).slice(2, 8)}`; +} + const hasUsageValues = ( usage: ReturnType, ): usage is NonNullable> => @@ -515,13 +519,15 @@ export async function runEmbeddedPiAgent( : null; if (contextOverflowError) { + const overflowDiagId = createCompactionDiagId(); const errorText = contextOverflowError.text; const msgCount = attempt.messagesSnapshot?.length ?? 0; log.warn( `[context-overflow-diag] sessionKey=${params.sessionKey ?? params.sessionId} ` + `provider=${provider}/${modelId} source=${contextOverflowError.source} ` + `messages=${msgCount} sessionFile=${params.sessionFile} ` + - `compactionAttempts=${overflowCompactionAttempts} error=${errorText.slice(0, 200)}`, + `diagId=${overflowDiagId} compactionAttempts=${overflowCompactionAttempts} ` + + `error=${errorText.slice(0, 200)}`, ); const isCompactionFailure = isCompactionFailureError(errorText); // Attempt auto-compaction on context overflow (not compaction_failure) @@ -529,6 +535,13 @@ export async function runEmbeddedPiAgent( !isCompactionFailure && overflowCompactionAttempts < MAX_OVERFLOW_COMPACTION_ATTEMPTS ) { + if (log.isEnabled("debug")) { + log.debug( + `[compaction-diag] decision diagId=${overflowDiagId} branch=compact ` + + `isCompactionFailure=${isCompactionFailure} hasOversizedToolResults=unknown ` + + `attempt=${overflowCompactionAttempts + 1} maxAttempts=${MAX_OVERFLOW_COMPACTION_ATTEMPTS}`, + ); + } overflowCompactionAttempts++; log.warn( `context overflow detected (attempt ${overflowCompactionAttempts}/${MAX_OVERFLOW_COMPACTION_ATTEMPTS}); attempting auto-compaction for ${provider}/${modelId}`, @@ -548,11 +561,16 @@ export async function runEmbeddedPiAgent( senderIsOwner: params.senderIsOwner, provider, model: modelId, + runId: params.runId, thinkLevel, reasoningLevel: params.reasoningLevel, bashElevated: params.bashElevated, extraSystemPrompt: params.extraSystemPrompt, ownerNumbers: params.ownerNumbers, + trigger: "overflow", + diagId: overflowDiagId, + attempt: overflowCompactionAttempts, + maxAttempts: MAX_OVERFLOW_COMPACTION_ATTEMPTS, }); if (compactResult.compacted) { autoCompactionCount += 1; @@ -576,6 +594,13 @@ export async function runEmbeddedPiAgent( : false; if (hasOversized) { + if (log.isEnabled("debug")) { + log.debug( + `[compaction-diag] decision diagId=${overflowDiagId} branch=truncate_tool_results ` + + `isCompactionFailure=${isCompactionFailure} hasOversizedToolResults=${hasOversized} ` + + `attempt=${overflowCompactionAttempts} maxAttempts=${MAX_OVERFLOW_COMPACTION_ATTEMPTS}`, + ); + } toolResultTruncationAttempted = true; log.warn( `[context-overflow-recovery] Attempting tool result truncation for ${provider}/${modelId} ` + @@ -598,8 +623,26 @@ export async function runEmbeddedPiAgent( log.warn( `[context-overflow-recovery] Tool result truncation did not help: ${truncResult.reason ?? "unknown"}`, ); + } else if (log.isEnabled("debug")) { + log.debug( + `[compaction-diag] decision diagId=${overflowDiagId} branch=give_up ` + + `isCompactionFailure=${isCompactionFailure} hasOversizedToolResults=${hasOversized} ` + + `attempt=${overflowCompactionAttempts} maxAttempts=${MAX_OVERFLOW_COMPACTION_ATTEMPTS}`, + ); } } + if ( + (isCompactionFailure || + overflowCompactionAttempts >= MAX_OVERFLOW_COMPACTION_ATTEMPTS || + toolResultTruncationAttempted) && + log.isEnabled("debug") + ) { + log.debug( + `[compaction-diag] decision diagId=${overflowDiagId} branch=give_up ` + + `isCompactionFailure=${isCompactionFailure} hasOversizedToolResults=unknown ` + + `attempt=${overflowCompactionAttempts} maxAttempts=${MAX_OVERFLOW_COMPACTION_ATTEMPTS}`, + ); + } const kind = isCompactionFailure ? "compaction_failure" : "context_overflow"; return { payloads: [ diff --git a/src/agents/pi-embedded-runner/run/attempt.ts b/src/agents/pi-embedded-runner/run/attempt.ts index 425a30a506d..7b91249a4bb 100644 --- a/src/agents/pi-embedded-runner/run/attempt.ts +++ b/src/agents/pi-embedded-runner/run/attempt.ts @@ -31,6 +31,7 @@ import { resolveOpenClawDocsPath } from "../../docs-path.js"; import { isTimeoutError } from "../../failover-error.js"; import { resolveModelAuthMode } from "../../model-auth.js"; import { resolveDefaultModelForAgent } from "../../model-selection.js"; +import { createOllamaStreamFn, OLLAMA_NATIVE_BASE_URL } from "../../ollama-stream.js"; import { isCloudCodeAssistFormatError, resolveBootstrapMaxChars, @@ -140,6 +141,69 @@ export function injectHistoryImagesIntoMessages( return didMutate; } +function summarizeMessagePayload(msg: AgentMessage): { textChars: number; imageBlocks: number } { + const content = (msg as { content?: unknown }).content; + if (typeof content === "string") { + return { textChars: content.length, imageBlocks: 0 }; + } + if (!Array.isArray(content)) { + return { textChars: 0, imageBlocks: 0 }; + } + + let textChars = 0; + let imageBlocks = 0; + for (const block of content) { + if (!block || typeof block !== "object") { + continue; + } + const typedBlock = block as { type?: unknown; text?: unknown }; + if (typedBlock.type === "image") { + imageBlocks++; + continue; + } + if (typeof typedBlock.text === "string") { + textChars += typedBlock.text.length; + } + } + + return { textChars, imageBlocks }; +} + +function summarizeSessionContext(messages: AgentMessage[]): { + roleCounts: string; + totalTextChars: number; + totalImageBlocks: number; + maxMessageTextChars: number; +} { + const roleCounts = new Map(); + let totalTextChars = 0; + let totalImageBlocks = 0; + let maxMessageTextChars = 0; + + for (const msg of messages) { + const role = typeof msg.role === "string" ? msg.role : "unknown"; + roleCounts.set(role, (roleCounts.get(role) ?? 0) + 1); + + const payload = summarizeMessagePayload(msg); + totalTextChars += payload.textChars; + totalImageBlocks += payload.imageBlocks; + if (payload.textChars > maxMessageTextChars) { + maxMessageTextChars = payload.textChars; + } + } + + return { + roleCounts: + [...roleCounts.entries()] + .toSorted((a, b) => a[0].localeCompare(b[0])) + .map(([role, count]) => `${role}:${count}`) + .join(",") || "none", + totalTextChars, + totalImageBlocks, + maxMessageTextChars, + }; +} + export async function runEmbeddedAttempt( params: EmbeddedRunAttemptParams, ): Promise { @@ -521,8 +585,21 @@ export async function runEmbeddedAttempt( workspaceDir: params.workspaceDir, }); - // Force a stable streamFn reference so vitest can reliably mock @mariozechner/pi-ai. - activeSession.agent.streamFn = streamSimple; + // Ollama native API: bypass SDK's streamSimple and use direct /api/chat calls + // for reliable streaming + tool calling support (#11828). + if (params.model.api === "ollama") { + // Use the resolved model baseUrl first so custom provider aliases work. + const providerConfig = params.config?.models?.providers?.[params.model.provider]; + const modelBaseUrl = + typeof params.model.baseUrl === "string" ? params.model.baseUrl.trim() : ""; + const providerBaseUrl = + typeof providerConfig?.baseUrl === "string" ? providerConfig.baseUrl.trim() : ""; + const ollamaBaseUrl = modelBaseUrl || providerBaseUrl || OLLAMA_NATIVE_BASE_URL; + activeSession.agent.streamFn = createOllamaStreamFn(ollamaBaseUrl); + } else { + // Force a stable streamFn reference so vitest can reliably mock @mariozechner/pi-ai. + activeSession.agent.streamFn = streamSimple; + } applyExtraParamsToAgent( activeSession.agent, @@ -749,6 +826,7 @@ export async function runEmbeddedAttempt( { agentId: hookAgentId, sessionKey: params.sessionKey, + sessionId: params.sessionId, workspaceDir: params.workspaceDir, messageProvider: params.messageProvider ?? undefined, }, @@ -825,6 +903,25 @@ export async function runEmbeddedAttempt( note: `images: prompt=${imageResult.images.length} history=${imageResult.historyImagesByIndex.size}`, }); + // Diagnostic: log context sizes before prompt to help debug early overflow errors. + if (log.isEnabled("debug")) { + const msgCount = activeSession.messages.length; + const systemLen = systemPromptText?.length ?? 0; + const promptLen = effectivePrompt.length; + const sessionSummary = summarizeSessionContext(activeSession.messages); + log.debug( + `[context-diag] pre-prompt: sessionKey=${params.sessionKey ?? params.sessionId} ` + + `messages=${msgCount} roleCounts=${sessionSummary.roleCounts} ` + + `historyTextChars=${sessionSummary.totalTextChars} ` + + `maxMessageTextChars=${sessionSummary.maxMessageTextChars} ` + + `historyImageBlocks=${sessionSummary.totalImageBlocks} ` + + `systemPromptChars=${systemLen} promptChars=${promptLen} ` + + `promptImages=${imageResult.images.length} ` + + `historyImageMessages=${imageResult.historyImagesByIndex.size} ` + + `provider=${params.provider}/${params.modelId} sessionFile=${params.sessionFile}`, + ); + } + // Only pass images option if there are actually images to pass // This avoids potential issues with models that don't expect the images parameter if (imageResult.images.length > 0) { @@ -890,6 +987,7 @@ export async function runEmbeddedAttempt( { agentId: hookAgentId, sessionKey: params.sessionKey, + sessionId: params.sessionId, workspaceDir: params.workspaceDir, messageProvider: params.messageProvider ?? undefined, }, diff --git a/src/agents/pi-embedded-runner/runs.ts b/src/agents/pi-embedded-runner/runs.ts index f5ca9721083..e0155874028 100644 --- a/src/agents/pi-embedded-runner/runs.ts +++ b/src/agents/pi-embedded-runner/runs.ts @@ -64,6 +64,10 @@ export function isEmbeddedPiRunStreaming(sessionId: string): boolean { return handle.isStreaming(); } +export function getActiveEmbeddedRunCount(): number { + return ACTIVE_EMBEDDED_RUNS.size; +} + export function waitForEmbeddedPiRunEnd(sessionId: string, timeoutMs = 15_000): Promise { if (!sessionId || !ACTIVE_EMBEDDED_RUNS.has(sessionId)) { return Promise.resolve(true); diff --git a/src/agents/pi-embedded-runner/utils.ts b/src/agents/pi-embedded-runner/utils.ts index 02daedec875..07fba6458c3 100644 --- a/src/agents/pi-embedded-runner/utils.ts +++ b/src/agents/pi-embedded-runner/utils.ts @@ -1,7 +1,5 @@ import type { ThinkingLevel } from "@mariozechner/pi-agent-core"; import type { ReasoningLevel, ThinkLevel } from "../../auto-reply/thinking.js"; -import type { OpenClawConfig } from "../../config/config.js"; -import type { ExecToolDefaults } from "../bash-tools.js"; export function mapThinkingLevel(level?: ThinkLevel): ThinkingLevel { // pi-agent-core supports "xhigh"; OpenClaw enables it for specific models. @@ -11,14 +9,6 @@ export function mapThinkingLevel(level?: ThinkLevel): ThinkingLevel { return level; } -export function resolveExecToolDefaults(config?: OpenClawConfig): ExecToolDefaults | undefined { - const tools = config?.tools; - if (!tools?.exec) { - return undefined; - } - return tools.exec; -} - export function describeUnknownError(error: unknown): string { if (error instanceof Error) { return error.message; diff --git a/src/agents/pi-embedded-subscribe.handlers.tools.ts b/src/agents/pi-embedded-subscribe.handlers.tools.ts index 3ab11f985f9..ba2151be8da 100644 --- a/src/agents/pi-embedded-subscribe.handlers.tools.ts +++ b/src/agents/pi-embedded-subscribe.handlers.tools.ts @@ -1,8 +1,5 @@ import type { AgentEvent } from "@mariozechner/pi-agent-core"; -import type { - PluginHookAfterToolCallEvent, - PluginHookBeforeToolCallEvent, -} from "../plugins/types.js"; +import type { PluginHookAfterToolCallEvent } from "../plugins/types.js"; import type { EmbeddedPiSubscribeContext } from "./pi-embedded-subscribe.handlers.types.js"; import { emitAgentEvent } from "../infra/agent-events.js"; import { getGlobalHookRunner } from "../plugins/hook-runner-global.js"; @@ -61,20 +58,6 @@ export async function handleToolExecutionStart( // Track start time and args for after_tool_call hook toolStartData.set(toolCallId, { startTime: Date.now(), args }); - // Call before_tool_call hook - const hookRunner = ctx.hookRunner ?? getGlobalHookRunner(); - if (hookRunner?.hasHooks?.("before_tool_call")) { - try { - const hookEvent: PluginHookBeforeToolCallEvent = { - toolName, - params: args && typeof args === "object" ? (args as Record) : {}, - }; - await hookRunner.runBeforeToolCall(hookEvent, { toolName }); - } catch (err) { - ctx.log.debug(`before_tool_call hook failed: tool=${toolName} error=${String(err)}`); - } - } - if (toolName === "read") { const record = args && typeof args === "object" ? (args as Record) : {}; const filePath = typeof record.path === "string" ? record.path.trim() : ""; diff --git a/src/agents/pi-tool-definition-adapter.after-tool-call.e2e.test.ts b/src/agents/pi-tool-definition-adapter.after-tool-call.e2e.test.ts index 7e7c74a35eb..cbcca9625b0 100644 --- a/src/agents/pi-tool-definition-adapter.after-tool-call.e2e.test.ts +++ b/src/agents/pi-tool-definition-adapter.after-tool-call.e2e.test.ts @@ -7,6 +7,8 @@ const hookMocks = vi.hoisted(() => ({ hasHooks: vi.fn(() => false), runAfterToolCall: vi.fn(async () => {}), }, + isToolWrappedWithBeforeToolCallHook: vi.fn(() => false), + consumeAdjustedParamsForToolCall: vi.fn(() => undefined), runBeforeToolCallHook: vi.fn(async ({ params }: { params: unknown }) => ({ blocked: false, params, @@ -18,6 +20,8 @@ vi.mock("../plugins/hook-runner-global.js", () => ({ })); vi.mock("./pi-tools.before-tool-call.js", () => ({ + consumeAdjustedParamsForToolCall: hookMocks.consumeAdjustedParamsForToolCall, + isToolWrappedWithBeforeToolCallHook: hookMocks.isToolWrappedWithBeforeToolCallHook, runBeforeToolCallHook: hookMocks.runBeforeToolCallHook, })); @@ -26,6 +30,10 @@ describe("pi tool definition adapter after_tool_call", () => { hookMocks.runner.hasHooks.mockReset(); hookMocks.runner.runAfterToolCall.mockReset(); hookMocks.runner.runAfterToolCall.mockResolvedValue(undefined); + hookMocks.isToolWrappedWithBeforeToolCallHook.mockReset(); + hookMocks.isToolWrappedWithBeforeToolCallHook.mockReturnValue(false); + hookMocks.consumeAdjustedParamsForToolCall.mockReset(); + hookMocks.consumeAdjustedParamsForToolCall.mockReturnValue(undefined); hookMocks.runBeforeToolCallHook.mockReset(); hookMocks.runBeforeToolCallHook.mockImplementation(async ({ params }) => ({ blocked: false, @@ -62,6 +70,38 @@ describe("pi tool definition adapter after_tool_call", () => { ); }); + it("uses wrapped-tool adjusted params for after_tool_call payload", async () => { + hookMocks.runner.hasHooks.mockImplementation((name: string) => name === "after_tool_call"); + hookMocks.isToolWrappedWithBeforeToolCallHook.mockReturnValue(true); + hookMocks.consumeAdjustedParamsForToolCall.mockReturnValue({ mode: "safe" }); + const tool = { + name: "read", + label: "Read", + description: "reads", + parameters: {}, + execute: vi.fn(async () => ({ content: [], details: { ok: true } })), + } satisfies AgentTool; + + const defs = toToolDefinitions([tool]); + const result = await defs[0].execute( + "call-ok-wrapped", + { path: "/tmp/file" }, + undefined, + undefined, + ); + + expect(result.details).toMatchObject({ ok: true }); + expect(hookMocks.runBeforeToolCallHook).not.toHaveBeenCalled(); + expect(hookMocks.runner.runAfterToolCall).toHaveBeenCalledWith( + { + toolName: "read", + params: { mode: "safe" }, + result, + }, + { toolName: "read" }, + ); + }); + it("dispatches after_tool_call once on adapter error with normalized tool name", async () => { hookMocks.runner.hasHooks.mockImplementation((name: string) => name === "after_tool_call"); const tool = { diff --git a/src/agents/pi-tool-definition-adapter.ts b/src/agents/pi-tool-definition-adapter.ts index 159b12cf3ca..ee02c2f9045 100644 --- a/src/agents/pi-tool-definition-adapter.ts +++ b/src/agents/pi-tool-definition-adapter.ts @@ -8,7 +8,11 @@ import type { ClientToolDefinition } from "./pi-embedded-runner/run/params.js"; import { logDebug, logError } from "../logger.js"; import { getGlobalHookRunner } from "../plugins/hook-runner-global.js"; import { isPlainObject } from "../utils.js"; -import { runBeforeToolCallHook } from "./pi-tools.before-tool-call.js"; +import { + consumeAdjustedParamsForToolCall, + isToolWrappedWithBeforeToolCallHook, + runBeforeToolCallHook, +} from "./pi-tools.before-tool-call.js"; import { normalizeToolName } from "./tool-policy.js"; import { jsonResult } from "./tools/common.js"; @@ -83,6 +87,7 @@ export function toToolDefinitions(tools: AnyAgentTool[]): ToolDefinition[] { return tools.map((tool) => { const name = tool.name || "tool"; const normalizedName = normalizeToolName(name); + const beforeHookWrapped = isToolWrappedWithBeforeToolCallHook(tool); return { name, label: tool.label ?? name, @@ -90,18 +95,23 @@ export function toToolDefinitions(tools: AnyAgentTool[]): ToolDefinition[] { parameters: tool.parameters, execute: async (...args: ToolExecuteArgs): Promise> => { const { toolCallId, params, onUpdate, signal } = splitToolExecuteArgs(args); + let executeParams = params; try { - // Call before_tool_call hook - const hookOutcome = await runBeforeToolCallHook({ - toolName: name, - params, - toolCallId, - }); - if (hookOutcome.blocked) { - throw new Error(hookOutcome.reason); + if (!beforeHookWrapped) { + const hookOutcome = await runBeforeToolCallHook({ + toolName: name, + params, + toolCallId, + }); + if (hookOutcome.blocked) { + throw new Error(hookOutcome.reason); + } + executeParams = hookOutcome.params; } - const adjustedParams = hookOutcome.params; - const result = await tool.execute(toolCallId, adjustedParams, signal, onUpdate); + const result = await tool.execute(toolCallId, executeParams, signal, onUpdate); + const afterParams = beforeHookWrapped + ? (consumeAdjustedParamsForToolCall(toolCallId) ?? executeParams) + : executeParams; // Call after_tool_call hook const hookRunner = getGlobalHookRunner(); @@ -110,7 +120,7 @@ export function toToolDefinitions(tools: AnyAgentTool[]): ToolDefinition[] { await hookRunner.runAfterToolCall( { toolName: name, - params: isPlainObject(adjustedParams) ? adjustedParams : {}, + params: isPlainObject(afterParams) ? afterParams : {}, result, }, { toolName: name }, @@ -134,6 +144,9 @@ export function toToolDefinitions(tools: AnyAgentTool[]): ToolDefinition[] { if (name === "AbortError") { throw err; } + if (beforeHookWrapped) { + consumeAdjustedParamsForToolCall(toolCallId); + } const described = describeToolExecutionError(err); if (described.stack && described.stack !== described.message) { logDebug(`tools: ${normalizedName} failed stack:\n${described.stack}`); diff --git a/src/agents/pi-tools-agent-config.e2e.test.ts b/src/agents/pi-tools-agent-config.e2e.test.ts index 012c7e30c37..69e05c60776 100644 --- a/src/agents/pi-tools-agent-config.e2e.test.ts +++ b/src/agents/pi-tools-agent-config.e2e.test.ts @@ -535,4 +535,59 @@ describe("Agent-specific tool filtering", () => { expect(result?.details.status).toBe("completed"); }); + + it("should apply agent-specific exec host defaults over global defaults", async () => { + const cfg: OpenClawConfig = { + tools: { + exec: { + host: "sandbox", + }, + }, + agents: { + list: [ + { + id: "main", + tools: { + exec: { + host: "gateway", + }, + }, + }, + { + id: "helper", + }, + ], + }, + }; + + const mainTools = createOpenClawCodingTools({ + config: cfg, + sessionKey: "agent:main:main", + workspaceDir: "/tmp/test-main-exec-defaults", + agentDir: "/tmp/agent-main-exec-defaults", + }); + const mainExecTool = mainTools.find((tool) => tool.name === "exec"); + expect(mainExecTool).toBeDefined(); + await expect( + mainExecTool!.execute("call-main", { + command: "echo done", + host: "sandbox", + }), + ).rejects.toThrow("exec host not allowed"); + + const helperTools = createOpenClawCodingTools({ + config: cfg, + sessionKey: "agent:helper:main", + workspaceDir: "/tmp/test-helper-exec-defaults", + agentDir: "/tmp/agent-helper-exec-defaults", + }); + const helperExecTool = helperTools.find((tool) => tool.name === "exec"); + expect(helperExecTool).toBeDefined(); + const helperResult = await helperExecTool!.execute("call-helper", { + command: "echo done", + host: "sandbox", + yieldMs: 10, + }); + expect(helperResult?.details.status).toBe("completed"); + }); }); diff --git a/src/agents/pi-tools.before-tool-call.e2e.test.ts b/src/agents/pi-tools.before-tool-call.e2e.test.ts index efc6c01104e..f6a81bf1fc3 100644 --- a/src/agents/pi-tools.before-tool-call.e2e.test.ts +++ b/src/agents/pi-tools.before-tool-call.e2e.test.ts @@ -1,6 +1,6 @@ import { beforeEach, describe, expect, it, vi } from "vitest"; import { getGlobalHookRunner } from "../plugins/hook-runner-global.js"; -import { toClientToolDefinitions } from "./pi-tool-definition-adapter.js"; +import { toClientToolDefinitions, toToolDefinitions } from "./pi-tool-definition-adapter.js"; import { wrapToolWithBeforeToolCallHook } from "./pi-tools.before-tool-call.js"; vi.mock("../plugins/hook-runner-global.js"); @@ -108,6 +108,44 @@ describe("before_tool_call hook integration", () => { }); }); +describe("before_tool_call hook deduplication (#15502)", () => { + let hookRunner: { + hasHooks: ReturnType; + runBeforeToolCall: ReturnType; + }; + + beforeEach(() => { + hookRunner = { + hasHooks: vi.fn(() => true), + runBeforeToolCall: vi.fn(async () => undefined), + }; + // oxlint-disable-next-line typescript/no-explicit-any + mockGetGlobalHookRunner.mockReturnValue(hookRunner as any); + }); + + it("fires hook exactly once when tool goes through wrap + toToolDefinitions", async () => { + const execute = vi.fn().mockResolvedValue({ content: [], details: { ok: true } }); + // oxlint-disable-next-line typescript/no-explicit-any + const baseTool = { name: "web_fetch", execute, description: "fetch", parameters: {} } as any; + + const wrapped = wrapToolWithBeforeToolCallHook(baseTool, { + agentId: "main", + sessionKey: "main", + }); + const [def] = toToolDefinitions([wrapped]); + + await def.execute( + "call-dedup", + { url: "https://example.com" }, + undefined, + undefined, + undefined, + ); + + expect(hookRunner.runBeforeToolCall).toHaveBeenCalledTimes(1); + }); +}); + describe("before_tool_call hook integration for client tools", () => { let hookRunner: { hasHooks: ReturnType; diff --git a/src/agents/pi-tools.before-tool-call.ts b/src/agents/pi-tools.before-tool-call.ts index aeca0af7540..26761f3127f 100644 --- a/src/agents/pi-tools.before-tool-call.ts +++ b/src/agents/pi-tools.before-tool-call.ts @@ -12,6 +12,9 @@ type HookContext = { type HookOutcome = { blocked: true; reason: string } | { blocked: false; params: unknown }; const log = createSubsystemLogger("agents/tools"); +const BEFORE_TOOL_CALL_WRAPPED = Symbol("beforeToolCallWrapped"); +const adjustedParamsByToolCallId = new Map(); +const MAX_TRACKED_ADJUSTED_PARAMS = 1024; export async function runBeforeToolCallHook(args: { toolName: string; @@ -71,7 +74,7 @@ export function wrapToolWithBeforeToolCallHook( return tool; } const toolName = tool.name || "tool"; - return { + const wrappedTool: AnyAgentTool = { ...tool, execute: async (toolCallId, params, signal, onUpdate) => { const outcome = await runBeforeToolCallHook({ @@ -83,12 +86,39 @@ export function wrapToolWithBeforeToolCallHook( if (outcome.blocked) { throw new Error(outcome.reason); } + if (toolCallId) { + adjustedParamsByToolCallId.set(toolCallId, outcome.params); + if (adjustedParamsByToolCallId.size > MAX_TRACKED_ADJUSTED_PARAMS) { + const oldest = adjustedParamsByToolCallId.keys().next().value; + if (oldest) { + adjustedParamsByToolCallId.delete(oldest); + } + } + } return await execute(toolCallId, outcome.params, signal, onUpdate); }, }; + Object.defineProperty(wrappedTool, BEFORE_TOOL_CALL_WRAPPED, { + value: true, + enumerable: false, + }); + return wrappedTool; +} + +export function isToolWrappedWithBeforeToolCallHook(tool: AnyAgentTool): boolean { + const taggedTool = tool as unknown as Record; + return taggedTool[BEFORE_TOOL_CALL_WRAPPED] === true; +} + +export function consumeAdjustedParamsForToolCall(toolCallId: string): unknown { + const params = adjustedParamsByToolCallId.get(toolCallId); + adjustedParamsByToolCallId.delete(toolCallId); + return params; } export const __testing = { + BEFORE_TOOL_CALL_WRAPPED, + adjustedParamsByToolCallId, runBeforeToolCallHook, isPlainObject, }; diff --git a/src/agents/pi-tools.ts b/src/agents/pi-tools.ts index d3118fbbcc2..26e16008c07 100644 --- a/src/agents/pi-tools.ts +++ b/src/agents/pi-tools.ts @@ -13,6 +13,7 @@ import { logWarn } from "../logger.js"; import { getPluginToolMeta } from "../plugins/tools.js"; import { isSubagentSessionKey } from "../routing/session-key.js"; import { resolveGatewayMessageChannel } from "../utils/message-channel.js"; +import { resolveAgentConfig } from "./agent-scope.js"; import { createApplyPatchTool } from "./apply-patch.js"; import { createExecTool, @@ -86,21 +87,25 @@ function isApplyPatchAllowedForModel(params: { }); } -function resolveExecConfig(cfg: OpenClawConfig | undefined) { +function resolveExecConfig(params: { cfg?: OpenClawConfig; agentId?: string }) { + const cfg = params.cfg; const globalExec = cfg?.tools?.exec; + const agentExec = + cfg && params.agentId ? resolveAgentConfig(cfg, params.agentId)?.tools?.exec : undefined; return { - host: globalExec?.host, - security: globalExec?.security, - ask: globalExec?.ask, - node: globalExec?.node, - pathPrepend: globalExec?.pathPrepend, - safeBins: globalExec?.safeBins, - backgroundMs: globalExec?.backgroundMs, - timeoutSec: globalExec?.timeoutSec, - approvalRunningNoticeMs: globalExec?.approvalRunningNoticeMs, - cleanupMs: globalExec?.cleanupMs, - notifyOnExit: globalExec?.notifyOnExit, - applyPatch: globalExec?.applyPatch, + host: agentExec?.host ?? globalExec?.host, + security: agentExec?.security ?? globalExec?.security, + ask: agentExec?.ask ?? globalExec?.ask, + node: agentExec?.node ?? globalExec?.node, + pathPrepend: agentExec?.pathPrepend ?? globalExec?.pathPrepend, + safeBins: agentExec?.safeBins ?? globalExec?.safeBins, + backgroundMs: agentExec?.backgroundMs ?? globalExec?.backgroundMs, + timeoutSec: agentExec?.timeoutSec ?? globalExec?.timeoutSec, + approvalRunningNoticeMs: + agentExec?.approvalRunningNoticeMs ?? globalExec?.approvalRunningNoticeMs, + cleanupMs: agentExec?.cleanupMs ?? globalExec?.cleanupMs, + notifyOnExit: agentExec?.notifyOnExit ?? globalExec?.notifyOnExit, + applyPatch: agentExec?.applyPatch ?? globalExec?.applyPatch, }; } @@ -231,7 +236,7 @@ export function createOpenClawCodingTools(options?: { sandbox?.tools, subagentPolicy, ]); - const execConfig = resolveExecConfig(options?.config); + const execConfig = resolveExecConfig({ cfg: options?.config, agentId }); const sandboxRoot = sandbox?.workspaceDir; const sandboxFsBridge = sandbox?.fsBridge; const allowWorkspaceWrites = sandbox?.workspaceAccess !== "ro"; diff --git a/src/agents/subagent-announce.format.e2e.test.ts b/src/agents/subagent-announce.format.e2e.test.ts index b1a0f6dd14a..882e85a767b 100644 --- a/src/agents/subagent-announce.format.e2e.test.ts +++ b/src/agents/subagent-announce.format.e2e.test.ts @@ -292,7 +292,7 @@ describe("subagent announce formatting", () => { lastChannel: "whatsapp", lastTo: "+1555", queueMode: "collect", - queueDebounceMs: 80, + queueDebounceMs: 0, }, }; @@ -327,7 +327,7 @@ describe("subagent announce formatting", () => { }), ]); - await new Promise((r) => setTimeout(r, 120)); + await expect.poll(() => agentSpy.mock.calls.length).toBe(2); expect(agentSpy).toHaveBeenCalledTimes(2); const accountIds = agentSpy.mock.calls.map( (call) => (call?.[0] as { params?: { accountId?: string } })?.params?.accountId, @@ -513,58 +513,4 @@ describe("subagent announce formatting", () => { expect(call?.params?.channel).toBe("bluebubbles"); expect(call?.params?.to).toBe("bluebubbles:chat_guid:123"); }); - - it("splits collect-mode announces when accountId differs", async () => { - const { runSubagentAnnounceFlow } = await import("./subagent-announce.js"); - embeddedRunMock.isEmbeddedPiRunActive.mockReturnValue(true); - embeddedRunMock.isEmbeddedPiRunStreaming.mockReturnValue(false); - sessionStore = { - "agent:main:main": { - sessionId: "session-789", - lastChannel: "whatsapp", - lastTo: "+1555", - queueMode: "collect", - queueDebounceMs: 0, - }, - }; - - await runSubagentAnnounceFlow({ - childSessionKey: "agent:main:subagent:test", - childRunId: "run-a", - requesterSessionKey: "main", - requesterOrigin: { accountId: "acct-a" }, - requesterDisplayKey: "main", - task: "do thing", - timeoutMs: 1000, - cleanup: "keep", - waitForCompletion: false, - startedAt: 10, - endedAt: 20, - outcome: { status: "ok" }, - }); - - await runSubagentAnnounceFlow({ - childSessionKey: "agent:main:subagent:test", - childRunId: "run-b", - requesterSessionKey: "main", - requesterOrigin: { accountId: "acct-b" }, - requesterDisplayKey: "main", - task: "do thing", - timeoutMs: 1000, - cleanup: "keep", - waitForCompletion: false, - startedAt: 10, - endedAt: 20, - outcome: { status: "ok" }, - }); - - await expect.poll(() => agentSpy.mock.calls.length).toBe(2); - - const accountIds = agentSpy.mock.calls.map( - (call) => (call[0] as { params?: Record }).params?.accountId, - ); - expect(accountIds).toContain("acct-a"); - expect(accountIds).toContain("acct-b"); - expect(agentSpy).toHaveBeenCalledTimes(2); - }); }); diff --git a/src/agents/subagent-announce.ts b/src/agents/subagent-announce.ts index 2bca43901b0..a3093c7b909 100644 --- a/src/agents/subagent-announce.ts +++ b/src/agents/subagent-announce.ts @@ -298,6 +298,7 @@ async function readLatestAssistantReplyWithRetry(params: { initialReply?: string; maxWaitMs: number; }): Promise { + const RETRY_INTERVAL_MS = 100; let reply = params.initialReply?.trim() ? params.initialReply : undefined; if (reply) { return reply; @@ -305,7 +306,7 @@ async function readLatestAssistantReplyWithRetry(params: { const deadline = Date.now() + Math.max(0, Math.min(params.maxWaitMs, 15_000)); while (Date.now() < deadline) { - await new Promise((resolve) => setTimeout(resolve, 300)); + await new Promise((resolve) => setTimeout(resolve, RETRY_INTERVAL_MS)); const latest = await readLatestAssistantReply({ sessionKey: params.sessionKey }); if (latest?.trim()) { return latest; diff --git a/src/agents/synthetic-models.ts b/src/agents/synthetic-models.ts index 9b924780586..5d820c8474b 100644 --- a/src/agents/synthetic-models.ts +++ b/src/agents/synthetic-models.ts @@ -155,6 +155,14 @@ export const SYNTHETIC_MODEL_CATALOG = [ contextWindow: 198000, maxTokens: 128000, }, + { + id: "hf:zai-org/GLM-5", + name: "GLM-5", + reasoning: true, + input: ["text", "image"], + contextWindow: 256000, + maxTokens: 128000, + }, { id: "hf:deepseek-ai/DeepSeek-V3", name: "DeepSeek V3", diff --git a/src/agents/tools/gateway-tool.ts b/src/agents/tools/gateway-tool.ts index 9560b323c4a..127fe1ff184 100644 --- a/src/agents/tools/gateway-tool.ts +++ b/src/agents/tools/gateway-tool.ts @@ -1,7 +1,7 @@ import { Type } from "@sinclair/typebox"; import type { OpenClawConfig } from "../../config/config.js"; -import { loadConfig, resolveConfigSnapshotHash } from "../../config/io.js"; -import { loadSessionStore, resolveStorePath } from "../../config/sessions.js"; +import { resolveConfigSnapshotHash } from "../../config/io.js"; +import { extractDeliveryInfo } from "../../config/sessions.js"; import { formatDoctorNonInteractiveHint, type RestartSentinelPayload, @@ -69,7 +69,7 @@ export function createGatewayTool(opts?: { label: "Gateway", name: "gateway", description: - "Restart, apply config, or update the gateway in-place (SIGUSR1). Use config.patch for safe partial config updates (merges with existing). Use config.apply only when replacing entire config. Both trigger restart after writing.", + "Restart, apply config, or update the gateway in-place (SIGUSR1). Use config.patch for safe partial config updates (merges with existing). Use config.apply only when replacing entire config. Both trigger restart after writing. Always pass a human-readable completion message via the `note` parameter so the system can deliver it to the user after restart.", parameters: GatewayToolSchema, execute: async (_toolCallId, args) => { const params = args as Record; @@ -93,34 +93,8 @@ export function createGatewayTool(opts?: { const note = typeof params.note === "string" && params.note.trim() ? params.note.trim() : undefined; // Extract channel + threadId for routing after restart - let deliveryContext: { channel?: string; to?: string; accountId?: string } | undefined; - let threadId: string | undefined; - if (sessionKey) { - const threadMarker = ":thread:"; - const threadIndex = sessionKey.lastIndexOf(threadMarker); - const baseSessionKey = threadIndex === -1 ? sessionKey : sessionKey.slice(0, threadIndex); - const threadIdRaw = - threadIndex === -1 ? undefined : sessionKey.slice(threadIndex + threadMarker.length); - threadId = threadIdRaw?.trim() || undefined; - try { - const cfg = loadConfig(); - const storePath = resolveStorePath(cfg.session?.store); - const store = loadSessionStore(storePath); - let entry = store[sessionKey]; - if (!entry?.deliveryContext && threadIndex !== -1 && baseSessionKey) { - entry = store[baseSessionKey]; - } - if (entry?.deliveryContext) { - deliveryContext = { - channel: entry.deliveryContext.channel, - to: entry.deliveryContext.to, - accountId: entry.deliveryContext.accountId, - }; - } - } catch { - // ignore: best-effort - } - } + // Supports both :thread: (most channels) and :topic: (Telegram) + const { deliveryContext, threadId } = extractDeliveryInfo(sessionKey); const payload: RestartSentinelPayload = { kind: "restart", status: "ok", diff --git a/src/agents/tools/web-fetch-utils.ts b/src/agents/tools/web-fetch-utils.ts index 5e0a248df92..09716e2cd46 100644 --- a/src/agents/tools/web-fetch-utils.ts +++ b/src/agents/tools/web-fetch-utils.ts @@ -1,5 +1,32 @@ export type ExtractMode = "markdown" | "text"; +let readabilityDepsPromise: + | Promise<{ + Readability: typeof import("@mozilla/readability").Readability; + parseHTML: typeof import("linkedom").parseHTML; + }> + | undefined; + +async function loadReadabilityDeps(): Promise<{ + Readability: typeof import("@mozilla/readability").Readability; + parseHTML: typeof import("linkedom").parseHTML; +}> { + if (!readabilityDepsPromise) { + readabilityDepsPromise = Promise.all([import("@mozilla/readability"), import("linkedom")]).then( + ([readability, linkedom]) => ({ + Readability: readability.Readability, + parseHTML: linkedom.parseHTML, + }), + ); + } + try { + return await readabilityDepsPromise; + } catch (error) { + readabilityDepsPromise = undefined; + throw error; + } +} + function decodeEntities(value: string): string { return value .replace(/ /gi, " ") @@ -94,10 +121,7 @@ export async function extractReadableContent(params: { return rendered; }; try { - const [{ Readability }, { parseHTML }] = await Promise.all([ - import("@mozilla/readability"), - import("linkedom"), - ]); + const { Readability, parseHTML } = await loadReadabilityDeps(); const { document } = parseHTML(params.html); try { (document as { baseURI?: string }).baseURI = params.url; diff --git a/src/agents/tools/web-fetch.cf-markdown.test.ts b/src/agents/tools/web-fetch.cf-markdown.test.ts index d73300681fc..a9602291d2e 100644 --- a/src/agents/tools/web-fetch.cf-markdown.test.ts +++ b/src/agents/tools/web-fetch.cf-markdown.test.ts @@ -1,9 +1,15 @@ import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; import * as ssrf from "../../infra/net/ssrf.js"; import * as logger from "../../logger.js"; +import { createWebFetchTool } from "./web-tools.js"; const lookupMock = vi.fn(); const resolvePinnedHostname = ssrf.resolvePinnedHostname; +const baseToolConfig = { + config: { + tools: { web: { fetch: { cacheTtlMinutes: 0, firecrawl: { enabled: false } } } }, + }, +} as const; function makeHeaders(map: Record): { get: (key: string) => string | null } { return { @@ -51,12 +57,7 @@ describe("web_fetch Cloudflare Markdown for Agents", () => { // @ts-expect-error mock fetch global.fetch = fetchSpy; - const { createWebFetchTool } = await import("./web-tools.js"); - const tool = createWebFetchTool({ - config: { - tools: { web: { fetch: { cacheTtlMinutes: 0, firecrawl: { enabled: false } } } }, - }, - }); + const tool = createWebFetchTool(baseToolConfig); await tool?.execute?.("call", { url: "https://example.com/page" }); @@ -71,12 +72,7 @@ describe("web_fetch Cloudflare Markdown for Agents", () => { // @ts-expect-error mock fetch global.fetch = fetchSpy; - const { createWebFetchTool } = await import("./web-tools.js"); - const tool = createWebFetchTool({ - config: { - tools: { web: { fetch: { cacheTtlMinutes: 0, firecrawl: { enabled: false } } } }, - }, - }); + const tool = createWebFetchTool(baseToolConfig); const result = await tool?.execute?.("call", { url: "https://example.com/cf" }); expect(result?.details).toMatchObject({ @@ -96,12 +92,7 @@ describe("web_fetch Cloudflare Markdown for Agents", () => { // @ts-expect-error mock fetch global.fetch = fetchSpy; - const { createWebFetchTool } = await import("./web-tools.js"); - const tool = createWebFetchTool({ - config: { - tools: { web: { fetch: { cacheTtlMinutes: 0, firecrawl: { enabled: false } } } }, - }, - }); + const tool = createWebFetchTool(baseToolConfig); const result = await tool?.execute?.("call", { url: "https://example.com/html" }); expect(result?.details?.extractor).not.toBe("cf-markdown"); @@ -116,12 +107,7 @@ describe("web_fetch Cloudflare Markdown for Agents", () => { // @ts-expect-error mock fetch global.fetch = fetchSpy; - const { createWebFetchTool } = await import("./web-tools.js"); - const tool = createWebFetchTool({ - config: { - tools: { web: { fetch: { cacheTtlMinutes: 0, firecrawl: { enabled: false } } } }, - }, - }); + const tool = createWebFetchTool(baseToolConfig); await tool?.execute?.("call", { url: "https://example.com/tokens/private?token=secret" }); @@ -142,12 +128,7 @@ describe("web_fetch Cloudflare Markdown for Agents", () => { // @ts-expect-error mock fetch global.fetch = fetchSpy; - const { createWebFetchTool } = await import("./web-tools.js"); - const tool = createWebFetchTool({ - config: { - tools: { web: { fetch: { cacheTtlMinutes: 0, firecrawl: { enabled: false } } } }, - }, - }); + const tool = createWebFetchTool(baseToolConfig); const result = await tool?.execute?.("call", { url: "https://example.com/text-mode", @@ -169,12 +150,7 @@ describe("web_fetch Cloudflare Markdown for Agents", () => { // @ts-expect-error mock fetch global.fetch = fetchSpy; - const { createWebFetchTool } = await import("./web-tools.js"); - const tool = createWebFetchTool({ - config: { - tools: { web: { fetch: { cacheTtlMinutes: 0, firecrawl: { enabled: false } } } }, - }, - }); + const tool = createWebFetchTool(baseToolConfig); await tool?.execute?.("call", { url: "https://example.com/no-tokens" }); diff --git a/src/agents/venice-models.ts b/src/agents/venice-models.ts index 32bd2f93b99..cff2e9d51cf 100644 --- a/src/agents/venice-models.ts +++ b/src/agents/venice-models.ts @@ -300,6 +300,11 @@ export function buildVeniceModelDefinition(entry: VeniceCatalogEntry): ModelDefi cost: VENICE_DEFAULT_COST, contextWindow: entry.contextWindow, maxTokens: entry.maxTokens, + // Avoid usage-only streaming chunks that can break OpenAI-compatible parsers. + // See: https://github.com/openclaw/openclaw/issues/15819 + compat: { + supportsUsageInStreaming: false, + }, }; } @@ -381,6 +386,10 @@ export async function discoverVeniceModels(): Promise { cost: VENICE_DEFAULT_COST, contextWindow: apiModel.model_spec.availableContextTokens || 128000, maxTokens: 8192, + // Avoid usage-only streaming chunks that can break OpenAI-compatible parsers. + compat: { + supportsUsageInStreaming: false, + }, }); } } diff --git a/src/agents/workspace.e2e.test.ts b/src/agents/workspace.e2e.test.ts index d4f842e6ea0..e34ff634a86 100644 --- a/src/agents/workspace.e2e.test.ts +++ b/src/agents/workspace.e2e.test.ts @@ -1,9 +1,14 @@ +import fs from "node:fs/promises"; import path from "node:path"; import { describe, expect, it } from "vitest"; import { makeTempWorkspace, writeWorkspaceFile } from "../test-helpers/workspace.js"; import { + DEFAULT_AGENTS_FILENAME, + DEFAULT_BOOTSTRAP_FILENAME, + DEFAULT_HEARTBEAT_FILENAME, DEFAULT_MEMORY_ALT_FILENAME, DEFAULT_MEMORY_FILENAME, + ensureAgentWorkspace, loadWorkspaceBootstrapFiles, resolveDefaultAgentWorkspaceDir, } from "./workspace.js"; @@ -19,6 +24,21 @@ describe("resolveDefaultAgentWorkspaceDir", () => { }); }); +describe("ensureAgentWorkspace", () => { + it("does not create HEARTBEAT.md during bootstrap file initialization", async () => { + const tempDir = await makeTempWorkspace("openclaw-workspace-init-"); + + const result = await ensureAgentWorkspace({ dir: tempDir, ensureBootstrapFiles: true }); + + await expect(fs.access(path.join(tempDir, DEFAULT_AGENTS_FILENAME))).resolves.toBeUndefined(); + await expect( + fs.access(path.join(tempDir, DEFAULT_BOOTSTRAP_FILENAME)), + ).resolves.toBeUndefined(); + await expect(fs.access(path.join(tempDir, DEFAULT_HEARTBEAT_FILENAME))).rejects.toThrow(); + expect("heartbeatPath" in result).toBe(false); + }); +}); + describe("loadWorkspaceBootstrapFiles", () => { it("includes MEMORY.md when present", async () => { const tempDir = await makeTempWorkspace("openclaw-workspace-"); diff --git a/src/agents/workspace.load-extra-bootstrap-files.test.ts b/src/agents/workspace.load-extra-bootstrap-files.test.ts new file mode 100644 index 00000000000..32586029c02 --- /dev/null +++ b/src/agents/workspace.load-extra-bootstrap-files.test.ts @@ -0,0 +1,53 @@ +import fs from "node:fs/promises"; +import path from "node:path"; +import { describe, expect, it } from "vitest"; +import { makeTempWorkspace } from "../test-helpers/workspace.js"; +import { loadExtraBootstrapFiles } from "./workspace.js"; + +describe("loadExtraBootstrapFiles", () => { + it("loads recognized bootstrap files from glob patterns", async () => { + const workspaceDir = await makeTempWorkspace("openclaw-extra-bootstrap-glob-"); + const packageDir = path.join(workspaceDir, "packages", "core"); + await fs.mkdir(packageDir, { recursive: true }); + await fs.writeFile(path.join(packageDir, "TOOLS.md"), "tools", "utf-8"); + await fs.writeFile(path.join(packageDir, "README.md"), "not bootstrap", "utf-8"); + + const files = await loadExtraBootstrapFiles(workspaceDir, ["packages/*/*"]); + + expect(files).toHaveLength(1); + expect(files[0]?.name).toBe("TOOLS.md"); + expect(files[0]?.content).toBe("tools"); + }); + + it("keeps path-traversal attempts outside workspace excluded", async () => { + const rootDir = await makeTempWorkspace("openclaw-extra-bootstrap-root-"); + const workspaceDir = path.join(rootDir, "workspace"); + const outsideDir = path.join(rootDir, "outside"); + await fs.mkdir(workspaceDir, { recursive: true }); + await fs.mkdir(outsideDir, { recursive: true }); + await fs.writeFile(path.join(outsideDir, "AGENTS.md"), "outside", "utf-8"); + + const files = await loadExtraBootstrapFiles(workspaceDir, ["../outside/AGENTS.md"]); + + expect(files).toHaveLength(0); + }); + + it("supports symlinked workspace roots with realpath checks", async () => { + if (process.platform === "win32") { + return; + } + + const rootDir = await makeTempWorkspace("openclaw-extra-bootstrap-symlink-"); + const realWorkspace = path.join(rootDir, "real-workspace"); + const linkedWorkspace = path.join(rootDir, "linked-workspace"); + await fs.mkdir(realWorkspace, { recursive: true }); + await fs.writeFile(path.join(realWorkspace, "AGENTS.md"), "linked agents", "utf-8"); + await fs.symlink(realWorkspace, linkedWorkspace, "dir"); + + const files = await loadExtraBootstrapFiles(linkedWorkspace, ["AGENTS.md"]); + + expect(files).toHaveLength(1); + expect(files[0]?.name).toBe("AGENTS.md"); + expect(files[0]?.content).toBe("linked agents"); + }); +}); diff --git a/src/agents/workspace.ts b/src/agents/workspace.ts index 486dff87cc0..db074d94303 100644 --- a/src/agents/workspace.ts +++ b/src/agents/workspace.ts @@ -93,6 +93,19 @@ export type WorkspaceBootstrapFile = { missing: boolean; }; +/** Set of recognized bootstrap filenames for runtime validation */ +const VALID_BOOTSTRAP_NAMES: ReadonlySet = new Set([ + DEFAULT_AGENTS_FILENAME, + DEFAULT_SOUL_FILENAME, + DEFAULT_TOOLS_FILENAME, + DEFAULT_IDENTITY_FILENAME, + DEFAULT_USER_FILENAME, + DEFAULT_HEARTBEAT_FILENAME, + DEFAULT_BOOTSTRAP_FILENAME, + DEFAULT_MEMORY_FILENAME, + DEFAULT_MEMORY_ALT_FILENAME, +]); + async function writeFileIfMissing(filePath: string, content: string) { try { await fs.writeFile(filePath, content, { @@ -160,7 +173,6 @@ export async function ensureAgentWorkspace(params?: { toolsPath?: string; identityPath?: string; userPath?: string; - heartbeatPath?: string; bootstrapPath?: string; }> { const rawDir = params?.dir?.trim() ? params.dir.trim() : DEFAULT_AGENT_WORKSPACE_DIR; @@ -176,11 +188,13 @@ export async function ensureAgentWorkspace(params?: { const toolsPath = path.join(dir, DEFAULT_TOOLS_FILENAME); const identityPath = path.join(dir, DEFAULT_IDENTITY_FILENAME); const userPath = path.join(dir, DEFAULT_USER_FILENAME); - const heartbeatPath = path.join(dir, DEFAULT_HEARTBEAT_FILENAME); + // HEARTBEAT.md is intentionally NOT created from template. + // Per docs: "If the file is missing, the heartbeat still runs and the model decides what to do." + // Creating it from template (which is effectively empty) would cause heartbeat to be skipped. const bootstrapPath = path.join(dir, DEFAULT_BOOTSTRAP_FILENAME); const isBrandNewWorkspace = await (async () => { - const paths = [agentsPath, soulPath, toolsPath, identityPath, userPath, heartbeatPath]; + const paths = [agentsPath, soulPath, toolsPath, identityPath, userPath]; const existing = await Promise.all( paths.map(async (p) => { try { @@ -199,7 +213,6 @@ export async function ensureAgentWorkspace(params?: { const toolsTemplate = await loadTemplate(DEFAULT_TOOLS_FILENAME); const identityTemplate = await loadTemplate(DEFAULT_IDENTITY_FILENAME); const userTemplate = await loadTemplate(DEFAULT_USER_FILENAME); - const heartbeatTemplate = await loadTemplate(DEFAULT_HEARTBEAT_FILENAME); const bootstrapTemplate = await loadTemplate(DEFAULT_BOOTSTRAP_FILENAME); await writeFileIfMissing(agentsPath, agentsTemplate); @@ -207,7 +220,6 @@ export async function ensureAgentWorkspace(params?: { await writeFileIfMissing(toolsPath, toolsTemplate); await writeFileIfMissing(identityPath, identityTemplate); await writeFileIfMissing(userPath, userTemplate); - await writeFileIfMissing(heartbeatPath, heartbeatTemplate); if (isBrandNewWorkspace) { await writeFileIfMissing(bootstrapPath, bootstrapTemplate); } @@ -220,7 +232,6 @@ export async function ensureAgentWorkspace(params?: { toolsPath, identityPath, userPath, - heartbeatPath, bootstrapPath, }; } @@ -329,3 +340,71 @@ export function filterBootstrapFilesForSession( } return files.filter((file) => SUBAGENT_BOOTSTRAP_ALLOWLIST.has(file.name)); } + +export async function loadExtraBootstrapFiles( + dir: string, + extraPatterns: string[], +): Promise { + if (!extraPatterns.length) { + return []; + } + const resolvedDir = resolveUserPath(dir); + let realResolvedDir = resolvedDir; + try { + realResolvedDir = await fs.realpath(resolvedDir); + } catch { + // Keep lexical root if realpath fails. + } + + // Resolve glob patterns into concrete file paths + const resolvedPaths = new Set(); + for (const pattern of extraPatterns) { + if (pattern.includes("*") || pattern.includes("?") || pattern.includes("{")) { + try { + const matches = fs.glob(pattern, { cwd: resolvedDir }); + for await (const m of matches) { + resolvedPaths.add(m); + } + } catch { + // glob not available or pattern error β€” fall back to literal + resolvedPaths.add(pattern); + } + } else { + resolvedPaths.add(pattern); + } + } + + const result: WorkspaceBootstrapFile[] = []; + for (const relPath of resolvedPaths) { + const filePath = path.resolve(resolvedDir, relPath); + // Guard against path traversal β€” resolved path must stay within workspace + if (!filePath.startsWith(resolvedDir + path.sep) && filePath !== resolvedDir) { + continue; + } + try { + // Resolve symlinks and verify the real path is still within workspace + const realFilePath = await fs.realpath(filePath); + if ( + !realFilePath.startsWith(realResolvedDir + path.sep) && + realFilePath !== realResolvedDir + ) { + continue; + } + // Only load files whose basename is a recognized bootstrap filename + const baseName = path.basename(relPath); + if (!VALID_BOOTSTRAP_NAMES.has(baseName)) { + continue; + } + const content = await fs.readFile(realFilePath, "utf-8"); + result.push({ + name: baseName as WorkspaceBootstrapFileName, + path: filePath, + content, + missing: false, + }); + } catch { + // Silently skip missing extra files + } + } + return result; +} diff --git a/src/auto-reply/dispatch.test.ts b/src/auto-reply/dispatch.test.ts new file mode 100644 index 00000000000..9e9630c406c --- /dev/null +++ b/src/auto-reply/dispatch.test.ts @@ -0,0 +1,91 @@ +import { describe, expect, it, vi } from "vitest"; +import type { OpenClawConfig } from "../config/config.js"; +import type { ReplyDispatcher } from "./reply/reply-dispatcher.js"; +import { dispatchInboundMessage, withReplyDispatcher } from "./dispatch.js"; +import { buildTestCtx } from "./reply/test-ctx.js"; + +function createDispatcher(record: string[]): ReplyDispatcher { + return { + sendToolResult: () => true, + sendBlockReply: () => true, + sendFinalReply: () => true, + getQueuedCounts: () => ({ tool: 0, block: 0, final: 0 }), + markComplete: () => { + record.push("markComplete"); + }, + waitForIdle: async () => { + record.push("waitForIdle"); + }, + }; +} + +describe("withReplyDispatcher", () => { + it("always marks complete and waits for idle after success", async () => { + const order: string[] = []; + const dispatcher = createDispatcher(order); + + const result = await withReplyDispatcher({ + dispatcher, + run: async () => { + order.push("run"); + return "ok"; + }, + onSettled: () => { + order.push("onSettled"); + }, + }); + + expect(result).toBe("ok"); + expect(order).toEqual(["run", "markComplete", "waitForIdle", "onSettled"]); + }); + + it("still drains dispatcher after run throws", async () => { + const order: string[] = []; + const dispatcher = createDispatcher(order); + const onSettled = vi.fn(() => { + order.push("onSettled"); + }); + + await expect( + withReplyDispatcher({ + dispatcher, + run: async () => { + order.push("run"); + throw new Error("boom"); + }, + onSettled, + }), + ).rejects.toThrow("boom"); + + expect(onSettled).toHaveBeenCalledTimes(1); + expect(order).toEqual(["run", "markComplete", "waitForIdle", "onSettled"]); + }); + + it("dispatchInboundMessage owns dispatcher lifecycle", async () => { + const order: string[] = []; + const dispatcher = { + sendToolResult: () => true, + sendBlockReply: () => true, + sendFinalReply: () => { + order.push("sendFinalReply"); + return true; + }, + getQueuedCounts: () => ({ tool: 0, block: 0, final: 0 }), + markComplete: () => { + order.push("markComplete"); + }, + waitForIdle: async () => { + order.push("waitForIdle"); + }, + } satisfies ReplyDispatcher; + + await dispatchInboundMessage({ + ctx: buildTestCtx(), + cfg: {} as OpenClawConfig, + dispatcher, + replyResolver: async () => ({ text: "ok" }), + }); + + expect(order).toEqual(["sendFinalReply", "markComplete", "waitForIdle"]); + }); +}); diff --git a/src/auto-reply/dispatch.ts b/src/auto-reply/dispatch.ts index d018623c7e0..54bf79a7bae 100644 --- a/src/auto-reply/dispatch.ts +++ b/src/auto-reply/dispatch.ts @@ -14,6 +14,24 @@ import { export type DispatchInboundResult = DispatchFromConfigResult; +export async function withReplyDispatcher(params: { + dispatcher: ReplyDispatcher; + run: () => Promise; + onSettled?: () => void | Promise; +}): Promise { + try { + return await params.run(); + } finally { + // Ensure dispatcher reservations are always released on every exit path. + params.dispatcher.markComplete(); + try { + await params.dispatcher.waitForIdle(); + } finally { + await params.onSettled?.(); + } + } +} + export async function dispatchInboundMessage(params: { ctx: MsgContext | FinalizedMsgContext; cfg: OpenClawConfig; @@ -22,12 +40,16 @@ export async function dispatchInboundMessage(params: { replyResolver?: typeof import("./reply.js").getReplyFromConfig; }): Promise { const finalized = finalizeInboundContext(params.ctx); - return await dispatchReplyFromConfig({ - ctx: finalized, - cfg: params.cfg, + return await withReplyDispatcher({ dispatcher: params.dispatcher, - replyOptions: params.replyOptions, - replyResolver: params.replyResolver, + run: () => + dispatchReplyFromConfig({ + ctx: finalized, + cfg: params.cfg, + dispatcher: params.dispatcher, + replyOptions: params.replyOptions, + replyResolver: params.replyResolver, + }), }); } @@ -41,20 +63,20 @@ export async function dispatchInboundMessageWithBufferedDispatcher(params: { const { dispatcher, replyOptions, markDispatchIdle } = createReplyDispatcherWithTyping( params.dispatcherOptions, ); - - const result = await dispatchInboundMessage({ - ctx: params.ctx, - cfg: params.cfg, - dispatcher, - replyResolver: params.replyResolver, - replyOptions: { - ...params.replyOptions, - ...replyOptions, - }, - }); - - markDispatchIdle(); - return result; + try { + return await dispatchInboundMessage({ + ctx: params.ctx, + cfg: params.cfg, + dispatcher, + replyResolver: params.replyResolver, + replyOptions: { + ...params.replyOptions, + ...replyOptions, + }, + }); + } finally { + markDispatchIdle(); + } } export async function dispatchInboundMessageWithDispatcher(params: { @@ -65,13 +87,11 @@ export async function dispatchInboundMessageWithDispatcher(params: { replyResolver?: typeof import("./reply.js").getReplyFromConfig; }): Promise { const dispatcher = createReplyDispatcher(params.dispatcherOptions); - const result = await dispatchInboundMessage({ + return await dispatchInboundMessage({ ctx: params.ctx, cfg: params.cfg, dispatcher, replyResolver: params.replyResolver, replyOptions: params.replyOptions, }); - await dispatcher.waitForIdle(); - return result; } diff --git a/src/auto-reply/heartbeat.test.ts b/src/auto-reply/heartbeat.test.ts index 5763d16261b..0506f08af3e 100644 --- a/src/auto-reply/heartbeat.test.ts +++ b/src/auto-reply/heartbeat.test.ts @@ -107,6 +107,62 @@ describe("stripHeartbeatToken", () => { didStrip: true, }); }); + + it("strips trailing punctuation only when directly after the token", () => { + // Token with trailing dot/exclamation/dashes β†’ should still strip + expect(stripHeartbeatToken(`${HEARTBEAT_TOKEN}.`, { mode: "heartbeat" })).toEqual({ + shouldSkip: true, + text: "", + didStrip: true, + }); + expect(stripHeartbeatToken(`${HEARTBEAT_TOKEN}!!!`, { mode: "heartbeat" })).toEqual({ + shouldSkip: true, + text: "", + didStrip: true, + }); + expect(stripHeartbeatToken(`${HEARTBEAT_TOKEN}---`, { mode: "heartbeat" })).toEqual({ + shouldSkip: true, + text: "", + didStrip: true, + }); + }); + + it("strips a sentence-ending token and keeps trailing punctuation", () => { + // Token appears at sentence end with trailing punctuation. + expect( + stripHeartbeatToken(`I should not respond ${HEARTBEAT_TOKEN}.`, { + mode: "message", + }), + ).toEqual({ + shouldSkip: false, + text: `I should not respond.`, + didStrip: true, + }); + }); + + it("strips sentence-ending token with emphasis punctuation in heartbeat mode", () => { + expect( + stripHeartbeatToken( + `There is nothing todo, so i should respond with ${HEARTBEAT_TOKEN} !!!`, + { + mode: "heartbeat", + }, + ), + ).toEqual({ + shouldSkip: true, + text: "", + didStrip: true, + }); + }); + + it("preserves trailing punctuation on text before the token", () => { + // Token at end, preceding text has its own punctuation β€” only the token is stripped + expect(stripHeartbeatToken(`All clear. ${HEARTBEAT_TOKEN}`, { mode: "message" })).toEqual({ + shouldSkip: false, + text: "All clear.", + didStrip: true, + }); + }); }); describe("isHeartbeatContentEffectivelyEmpty", () => { diff --git a/src/auto-reply/heartbeat.ts b/src/auto-reply/heartbeat.ts index 4f4ef22aa79..4141d180f67 100644 --- a/src/auto-reply/heartbeat.ts +++ b/src/auto-reply/heartbeat.ts @@ -1,3 +1,4 @@ +import { escapeRegExp } from "../utils.js"; import { HEARTBEAT_TOKEN } from "./tokens.js"; // Default heartbeat prompt (used when config.agents.defaults.heartbeat.prompt is unset). @@ -65,6 +66,9 @@ function stripTokenAtEdges(raw: string): { text: string; didStrip: boolean } { } const token = HEARTBEAT_TOKEN; + const tokenAtEndWithOptionalTrailingPunctuation = new RegExp( + `${escapeRegExp(token)}[^\\w]{0,4}$`, + ); if (!text.includes(token)) { return { text, didStrip: false }; } @@ -81,9 +85,19 @@ function stripTokenAtEdges(raw: string): { text: string; didStrip: boolean } { changed = true; continue; } - if (next.endsWith(token)) { - const before = next.slice(0, Math.max(0, next.length - token.length)); - text = before.trimEnd(); + // Strip the token when it appears at the end of the text. + // Also strip up to 4 trailing non-word characters the model may have appended + // (e.g. ".", "!!!", "---"). Keep trailing punctuation only when real + // sentence text exists before the token. + if (tokenAtEndWithOptionalTrailingPunctuation.test(next)) { + const idx = next.lastIndexOf(token); + const before = next.slice(0, idx).trimEnd(); + if (!before) { + text = ""; + } else { + const after = next.slice(idx + token.length).trimStart(); + text = `${before}${after}`.trimEnd(); + } didStrip = true; changed = true; } diff --git a/src/auto-reply/reply.block-streaming.test.ts b/src/auto-reply/reply.block-streaming.test.ts index 5a1f97d1d4d..0a5cb742315 100644 --- a/src/auto-reply/reply.block-streaming.test.ts +++ b/src/auto-reply/reply.block-streaming.test.ts @@ -1,6 +1,7 @@ +import fs from "node:fs/promises"; +import os from "node:os"; import path from "node:path"; -import { beforeEach, describe, expect, it, vi } from "vitest"; -import { withTempHome as withTempHomeBase } from "../../test/helpers/temp-home.js"; +import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; import { loadModelCatalog } from "../agents/model-catalog.js"; import { getReplyFromConfig } from "./reply.js"; @@ -22,11 +23,74 @@ vi.mock("../agents/model-catalog.js", () => ({ loadModelCatalog: vi.fn(), })); +type HomeEnvSnapshot = { + HOME: string | undefined; + USERPROFILE: string | undefined; + HOMEDRIVE: string | undefined; + HOMEPATH: string | undefined; + OPENCLAW_STATE_DIR: string | undefined; +}; + +function snapshotHomeEnv(): HomeEnvSnapshot { + return { + HOME: process.env.HOME, + USERPROFILE: process.env.USERPROFILE, + HOMEDRIVE: process.env.HOMEDRIVE, + HOMEPATH: process.env.HOMEPATH, + OPENCLAW_STATE_DIR: process.env.OPENCLAW_STATE_DIR, + }; +} + +function restoreHomeEnv(snapshot: HomeEnvSnapshot) { + for (const [key, value] of Object.entries(snapshot)) { + if (value === undefined) { + delete process.env[key]; + } else { + process.env[key] = value; + } + } +} + +let fixtureRoot = ""; +let caseId = 0; + async function withTempHome(fn: (home: string) => Promise): Promise { - return withTempHomeBase(fn, { prefix: "openclaw-stream-" }); + const home = path.join(fixtureRoot, `case-${++caseId}`); + await fs.mkdir(path.join(home, ".openclaw", "agents", "main", "sessions"), { recursive: true }); + const envSnapshot = snapshotHomeEnv(); + process.env.HOME = home; + process.env.USERPROFILE = home; + process.env.OPENCLAW_STATE_DIR = path.join(home, ".openclaw"); + + if (process.platform === "win32") { + const match = home.match(/^([A-Za-z]:)(.*)$/); + if (match) { + process.env.HOMEDRIVE = match[1]; + process.env.HOMEPATH = match[2] || "\\"; + } + } + + try { + return await fn(home); + } finally { + restoreHomeEnv(envSnapshot); + } } describe("block streaming", () => { + beforeAll(async () => { + fixtureRoot = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-stream-")); + }); + + afterAll(async () => { + await fs.rm(fixtureRoot, { + recursive: true, + force: true, + maxRetries: 10, + retryDelay: 50, + }); + }); + beforeEach(() => { piEmbeddedMock.abortEmbeddedPiRun.mockReset().mockReturnValue(false); piEmbeddedMock.queueEmbeddedPiMessage.mockReset().mockReturnValue(false); @@ -39,78 +103,20 @@ describe("block streaming", () => { ]); }); - async function waitForCalls(fn: () => number, calls: number) { - const deadline = Date.now() + 5000; - while (fn() < calls) { - if (Date.now() > deadline) { - throw new Error(`Expected ${calls} call(s), got ${fn()}`); - } - await new Promise((resolve) => setTimeout(resolve, 5)); - } - } - - it("waits for block replies before returning final payloads", async () => { + it("handles ordering, timeout fallback, and telegram streamMode block", async () => { await withTempHome(async (home) => { let releaseTyping: (() => void) | undefined; const typingGate = new Promise((resolve) => { releaseTyping = resolve; }); - const onReplyStart = vi.fn(() => typingGate); - const onBlockReply = vi.fn().mockResolvedValue(undefined); - - const impl = async (params: RunEmbeddedPiAgentParams) => { - void params.onBlockReply?.({ text: "hello" }); - return { - payloads: [{ text: "hello" }], - meta: { - durationMs: 5, - agentMeta: { sessionId: "s", provider: "p", model: "m" }, - }, - }; - }; - piEmbeddedMock.runEmbeddedPiAgent.mockImplementation(impl); - - const replyPromise = getReplyFromConfig( - { - Body: "ping", - From: "+1004", - To: "+2000", - MessageSid: "msg-123", - Provider: "discord", - }, - { - onReplyStart, - onBlockReply, - disableBlockStreaming: false, - }, - { - agents: { - defaults: { - model: "anthropic/claude-opus-4-5", - workspace: path.join(home, "openclaw"), - }, - }, - channels: { whatsapp: { allowFrom: ["*"] } }, - session: { store: path.join(home, "sessions.json") }, - }, - ); - - await waitForCalls(() => onReplyStart.mock.calls.length, 1); - releaseTyping?.(); - - const res = await replyPromise; - expect(res).toBeUndefined(); - expect(onBlockReply).toHaveBeenCalledTimes(1); - }); - }); - - it("preserves block reply ordering when typing start is slow", async () => { - await withTempHome(async (home) => { - let releaseTyping: (() => void) | undefined; - const typingGate = new Promise((resolve) => { - releaseTyping = resolve; + let resolveOnReplyStart: (() => void) | undefined; + const onReplyStartCalled = new Promise((resolve) => { + resolveOnReplyStart = resolve; + }); + const onReplyStart = vi.fn(() => { + resolveOnReplyStart?.(); + return typingGate; }); - const onReplyStart = vi.fn(() => typingGate); const seen: string[] = []; const onBlockReply = vi.fn(async (payload) => { seen.push(payload.text ?? ""); @@ -134,7 +140,7 @@ describe("block streaming", () => { Body: "ping", From: "+1004", To: "+2000", - MessageSid: "msg-125", + MessageSid: "msg-123", Provider: "telegram", }, { @@ -154,64 +160,15 @@ describe("block streaming", () => { }, ); - await waitForCalls(() => onReplyStart.mock.calls.length, 1); + await onReplyStartCalled; releaseTyping?.(); const res = await replyPromise; expect(res).toBeUndefined(); expect(seen).toEqual(["first\n\nsecond"]); - }); - }); - it("drops final payloads when block replies streamed", async () => { - await withTempHome(async (home) => { - const onBlockReply = vi.fn().mockResolvedValue(undefined); - - const impl = async (params: RunEmbeddedPiAgentParams) => { - void params.onBlockReply?.({ text: "chunk-1" }); - return { - payloads: [{ text: "chunk-1\nchunk-2" }], - meta: { - durationMs: 5, - agentMeta: { sessionId: "s", provider: "p", model: "m" }, - }, - }; - }; - piEmbeddedMock.runEmbeddedPiAgent.mockImplementation(impl); - - const res = await getReplyFromConfig( - { - Body: "ping", - From: "+1004", - To: "+2000", - MessageSid: "msg-124", - Provider: "discord", - }, - { - onBlockReply, - disableBlockStreaming: false, - }, - { - agents: { - defaults: { - model: "anthropic/claude-opus-4-5", - workspace: path.join(home, "openclaw"), - }, - }, - channels: { whatsapp: { allowFrom: ["*"] } }, - session: { store: path.join(home, "sessions.json") }, - }, - ); - - expect(res).toBeUndefined(); - expect(onBlockReply).toHaveBeenCalledTimes(1); - }); - }); - - it("falls back to final payloads when block reply send times out", async () => { - await withTempHome(async (home) => { let sawAbort = false; - const onBlockReply = vi.fn((_, context) => { + const onBlockReplyTimeout = vi.fn((_, context) => { return new Promise((resolve) => { context?.abortSignal?.addEventListener( "abort", @@ -224,7 +181,7 @@ describe("block streaming", () => { }); }); - const impl = async (params: RunEmbeddedPiAgentParams) => { + const timeoutImpl = async (params: RunEmbeddedPiAgentParams) => { void params.onBlockReply?.({ text: "streamed" }); return { payloads: [{ text: "final" }], @@ -234,9 +191,9 @@ describe("block streaming", () => { }, }; }; - piEmbeddedMock.runEmbeddedPiAgent.mockImplementation(impl); + piEmbeddedMock.runEmbeddedPiAgent.mockImplementation(timeoutImpl); - const replyPromise = getReplyFromConfig( + const timeoutReplyPromise = getReplyFromConfig( { Body: "ping", From: "+1004", @@ -245,8 +202,8 @@ describe("block streaming", () => { Provider: "telegram", }, { - onBlockReply, - blockReplyTimeoutMs: 10, + onBlockReply: onBlockReplyTimeout, + blockReplyTimeoutMs: 1, disableBlockStreaming: false, }, { @@ -261,35 +218,29 @@ describe("block streaming", () => { }, ); - const res = await replyPromise; - expect(res).toMatchObject({ text: "final" }); + const timeoutRes = await timeoutReplyPromise; + expect(timeoutRes).toMatchObject({ text: "final" }); expect(sawAbort).toBe(true); - }); - }); - it("does not enable block streaming for telegram streamMode block", async () => { - await withTempHome(async (home) => { - const onBlockReply = vi.fn().mockResolvedValue(undefined); - - const impl = async () => ({ + const onBlockReplyStreamMode = vi.fn().mockResolvedValue(undefined); + piEmbeddedMock.runEmbeddedPiAgent.mockImplementation(async () => ({ payloads: [{ text: "final" }], meta: { durationMs: 5, agentMeta: { sessionId: "s", provider: "p", model: "m" }, }, - }); - piEmbeddedMock.runEmbeddedPiAgent.mockImplementation(impl); + })); - const res = await getReplyFromConfig( + const resStreamMode = await getReplyFromConfig( { Body: "ping", From: "+1004", To: "+2000", - MessageSid: "msg-126", + MessageSid: "msg-127", Provider: "telegram", }, { - onBlockReply, + onBlockReply: onBlockReplyStreamMode, }, { agents: { @@ -303,8 +254,8 @@ describe("block streaming", () => { }, ); - expect(res?.text).toBe("final"); - expect(onBlockReply).not.toHaveBeenCalled(); + expect(resStreamMode?.text).toBe("final"); + expect(onBlockReplyStreamMode).not.toHaveBeenCalled(); }); }); }); diff --git a/src/auto-reply/reply.queue.test.ts b/src/auto-reply/reply.queue.test.ts deleted file mode 100644 index 2af49458bf0..00000000000 --- a/src/auto-reply/reply.queue.test.ts +++ /dev/null @@ -1,149 +0,0 @@ -import path from "node:path"; -import { afterEach, describe, expect, it, vi } from "vitest"; -import { pollUntil } from "../../test/helpers/poll.js"; -import { withTempHome as withTempHomeBase } from "../../test/helpers/temp-home.js"; -import { - isEmbeddedPiRunActive, - isEmbeddedPiRunStreaming, - runEmbeddedPiAgent, -} from "../agents/pi-embedded.js"; -import { getReplyFromConfig } from "./reply.js"; - -vi.mock("../agents/pi-embedded.js", () => ({ - abortEmbeddedPiRun: vi.fn().mockReturnValue(false), - runEmbeddedPiAgent: vi.fn(), - queueEmbeddedPiMessage: vi.fn().mockReturnValue(false), - resolveEmbeddedSessionLane: (key: string) => `session:${key.trim() || "main"}`, - isEmbeddedPiRunActive: vi.fn().mockReturnValue(false), - isEmbeddedPiRunStreaming: vi.fn().mockReturnValue(false), -})); - -function makeResult(text: string) { - return { - payloads: [{ text }], - meta: { - durationMs: 5, - agentMeta: { sessionId: "s", provider: "p", model: "m" }, - }, - }; -} - -async function withTempHome(fn: (home: string) => Promise): Promise { - return withTempHomeBase( - async (home) => { - vi.mocked(runEmbeddedPiAgent).mockReset(); - return await fn(home); - }, - { prefix: "openclaw-queue-" }, - ); -} - -function makeCfg(home: string, queue?: Record) { - return { - agents: { - defaults: { - model: "anthropic/claude-opus-4-5", - workspace: path.join(home, "openclaw"), - }, - }, - channels: { whatsapp: { allowFrom: ["*"] } }, - session: { store: path.join(home, "sessions.json") }, - messages: queue ? { queue } : undefined, - }; -} - -describe("queue followups", () => { - afterEach(() => { - vi.useRealTimers(); - }); - - it("collects queued messages and drains after run completes", async () => { - vi.useFakeTimers(); - await withTempHome(async (home) => { - const prompts: string[] = []; - vi.mocked(runEmbeddedPiAgent).mockImplementation(async (params) => { - prompts.push(params.prompt); - if (params.prompt.includes("[Queued messages while agent was busy]")) { - return makeResult("followup"); - } - return makeResult("main"); - }); - - vi.mocked(isEmbeddedPiRunActive).mockReturnValue(true); - vi.mocked(isEmbeddedPiRunStreaming).mockReturnValue(true); - - const cfg = makeCfg(home, { - mode: "collect", - debounceMs: 200, - cap: 10, - drop: "summarize", - }); - - const first = await getReplyFromConfig( - { Body: "first", From: "+1001", To: "+2000", MessageSid: "m-1" }, - {}, - cfg, - ); - expect(first).toBeUndefined(); - expect(runEmbeddedPiAgent).not.toHaveBeenCalled(); - - vi.mocked(isEmbeddedPiRunActive).mockReturnValue(false); - vi.mocked(isEmbeddedPiRunStreaming).mockReturnValue(false); - - const second = await getReplyFromConfig( - { Body: "second", From: "+1001", To: "+2000" }, - {}, - cfg, - ); - - const secondText = Array.isArray(second) ? second[0]?.text : second?.text; - expect(secondText).toBe("main"); - - await vi.advanceTimersByTimeAsync(500); - await Promise.resolve(); - - expect(runEmbeddedPiAgent).toHaveBeenCalledTimes(2); - const queuedPrompt = prompts.find((p) => - p.includes("[Queued messages while agent was busy]"), - ); - expect(queuedPrompt).toBeTruthy(); - // Message id hints are no longer exposed to the model prompt. - expect(queuedPrompt).toContain("Queued #1"); - expect(queuedPrompt).toContain("first"); - expect(queuedPrompt).not.toContain("[message_id:"); - }); - }); - - it("summarizes dropped followups when cap is exceeded", async () => { - await withTempHome(async (home) => { - const prompts: string[] = []; - vi.mocked(runEmbeddedPiAgent).mockImplementation(async (params) => { - prompts.push(params.prompt); - return makeResult("ok"); - }); - - vi.mocked(isEmbeddedPiRunActive).mockReturnValue(true); - vi.mocked(isEmbeddedPiRunStreaming).mockReturnValue(false); - - const cfg = makeCfg(home, { - mode: "followup", - debounceMs: 0, - cap: 1, - drop: "summarize", - }); - - await getReplyFromConfig({ Body: "one", From: "+1002", To: "+2000" }, {}, cfg); - await getReplyFromConfig({ Body: "two", From: "+1002", To: "+2000" }, {}, cfg); - - vi.mocked(isEmbeddedPiRunActive).mockReturnValue(false); - await getReplyFromConfig({ Body: "three", From: "+1002", To: "+2000" }, {}, cfg); - - await pollUntil( - async () => (prompts.some((p) => p.includes("[Queue overflow]")) ? true : null), - { timeoutMs: 2000 }, - ); - - expect(prompts.some((p) => p.includes("[Queue overflow]"))).toBe(true); - }); - }); -}); diff --git a/src/auto-reply/reply.raw-body.test.ts b/src/auto-reply/reply.raw-body.test.ts index 38c8b30e218..4adead48907 100644 --- a/src/auto-reply/reply.raw-body.test.ts +++ b/src/auto-reply/reply.raw-body.test.ts @@ -1,7 +1,7 @@ import fs from "node:fs/promises"; +import os from "node:os"; import path from "node:path"; -import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; -import { withTempHome as withTempHomeBase } from "../../test/helpers/temp-home.js"; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; import { loadModelCatalog } from "../agents/model-catalog.js"; import { runEmbeddedPiAgent } from "../agents/pi-embedded.js"; import { saveSessionStore } from "../config/sessions.js"; @@ -19,22 +19,75 @@ vi.mock("../agents/model-catalog.js", () => ({ loadModelCatalog: vi.fn(), })); +type HomeEnvSnapshot = { + HOME: string | undefined; + USERPROFILE: string | undefined; + HOMEDRIVE: string | undefined; + HOMEPATH: string | undefined; + OPENCLAW_STATE_DIR: string | undefined; + OPENCLAW_AGENT_DIR: string | undefined; + PI_CODING_AGENT_DIR: string | undefined; +}; + +function snapshotHomeEnv(): HomeEnvSnapshot { + return { + HOME: process.env.HOME, + USERPROFILE: process.env.USERPROFILE, + HOMEDRIVE: process.env.HOMEDRIVE, + HOMEPATH: process.env.HOMEPATH, + OPENCLAW_STATE_DIR: process.env.OPENCLAW_STATE_DIR, + OPENCLAW_AGENT_DIR: process.env.OPENCLAW_AGENT_DIR, + PI_CODING_AGENT_DIR: process.env.PI_CODING_AGENT_DIR, + }; +} + +function restoreHomeEnv(snapshot: HomeEnvSnapshot) { + for (const [key, value] of Object.entries(snapshot)) { + if (value === undefined) { + delete process.env[key]; + } else { + process.env[key] = value; + } + } +} + +let fixtureRoot = ""; +let caseId = 0; + async function withTempHome(fn: (home: string) => Promise): Promise { - return withTempHomeBase( - async (home) => { - return await fn(home); - }, - { - env: { - OPENCLAW_AGENT_DIR: (home) => path.join(home, ".openclaw", "agent"), - PI_CODING_AGENT_DIR: (home) => path.join(home, ".openclaw", "agent"), - }, - prefix: "openclaw-rawbody-", - }, - ); + const home = path.join(fixtureRoot, `case-${++caseId}`); + await fs.mkdir(path.join(home, ".openclaw", "agents", "main", "sessions"), { recursive: true }); + const envSnapshot = snapshotHomeEnv(); + process.env.HOME = home; + process.env.USERPROFILE = home; + process.env.OPENCLAW_STATE_DIR = path.join(home, ".openclaw"); + process.env.OPENCLAW_AGENT_DIR = path.join(home, ".openclaw", "agent"); + process.env.PI_CODING_AGENT_DIR = path.join(home, ".openclaw", "agent"); + + if (process.platform === "win32") { + const match = home.match(/^([A-Za-z]:)(.*)$/); + if (match) { + process.env.HOMEDRIVE = match[1]; + process.env.HOMEPATH = match[2] || "\\"; + } + } + + try { + return await fn(home); + } finally { + restoreHomeEnv(envSnapshot); + } } describe("RawBody directive parsing", () => { + beforeAll(async () => { + fixtureRoot = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-rawbody-")); + }); + + afterAll(async () => { + await fs.rm(fixtureRoot, { recursive: true, force: true }); + }); + beforeEach(() => { vi.mocked(runEmbeddedPiAgent).mockReset(); vi.mocked(loadModelCatalog).mockResolvedValue([ @@ -46,151 +99,7 @@ describe("RawBody directive parsing", () => { vi.clearAllMocks(); }); - it("/model, /think, /verbose directives detected from RawBody even when Body has structural wrapper", async () => { - await withTempHome(async (home) => { - vi.mocked(runEmbeddedPiAgent).mockReset(); - - const groupMessageCtx = { - Body: `[Chat messages since your last reply - for context]\\n[WhatsApp ...] Someone: hello\\n\\n[Current message - respond to this]\\n[WhatsApp ...] Jake: /think:high\\n[from: Jake McInteer (+6421807830)]`, - RawBody: "/think:high", - From: "+1222", - To: "+1222", - ChatType: "group", - CommandAuthorized: true, - }; - - const res = await getReplyFromConfig( - groupMessageCtx, - {}, - { - agents: { - defaults: { - model: "anthropic/claude-opus-4-5", - workspace: path.join(home, "openclaw"), - }, - }, - channels: { whatsapp: { allowFrom: ["*"] } }, - session: { store: path.join(home, "sessions.json") }, - }, - ); - - const text = Array.isArray(res) ? res[0]?.text : res?.text; - expect(text).toContain("Thinking level set to high."); - expect(runEmbeddedPiAgent).not.toHaveBeenCalled(); - }); - }); - - it("/model status detected from RawBody", async () => { - await withTempHome(async (home) => { - vi.mocked(runEmbeddedPiAgent).mockReset(); - - const groupMessageCtx = { - Body: `[Context]\nJake: /model status\n[from: Jake]`, - RawBody: "/model status", - From: "+1222", - To: "+1222", - ChatType: "group", - CommandAuthorized: true, - }; - - const res = await getReplyFromConfig( - groupMessageCtx, - {}, - { - agents: { - defaults: { - model: "anthropic/claude-opus-4-5", - workspace: path.join(home, "openclaw"), - models: { - "anthropic/claude-opus-4-5": {}, - }, - }, - }, - channels: { whatsapp: { allowFrom: ["*"] } }, - session: { store: path.join(home, "sessions.json") }, - }, - ); - - const text = Array.isArray(res) ? res[0]?.text : res?.text; - expect(text).toContain("anthropic/claude-opus-4-5"); - expect(runEmbeddedPiAgent).not.toHaveBeenCalled(); - }); - }); - - it("CommandBody is honored when RawBody is missing", async () => { - await withTempHome(async (home) => { - vi.mocked(runEmbeddedPiAgent).mockReset(); - - const groupMessageCtx = { - Body: `[Context]\nJake: /verbose on\n[from: Jake]`, - CommandBody: "/verbose on", - From: "+1222", - To: "+1222", - ChatType: "group", - CommandAuthorized: true, - }; - - const res = await getReplyFromConfig( - groupMessageCtx, - {}, - { - agents: { - defaults: { - model: "anthropic/claude-opus-4-5", - workspace: path.join(home, "openclaw"), - }, - }, - channels: { whatsapp: { allowFrom: ["*"] } }, - session: { store: path.join(home, "sessions.json") }, - }, - ); - - const text = Array.isArray(res) ? res[0]?.text : res?.text; - expect(text).toContain("Verbose logging enabled."); - expect(runEmbeddedPiAgent).not.toHaveBeenCalled(); - }); - }); - - it("Integration: WhatsApp group message with structural wrapper and RawBody command", async () => { - await withTempHome(async (home) => { - vi.mocked(runEmbeddedPiAgent).mockReset(); - - const groupMessageCtx = { - Body: `[Chat messages since your last reply - for context]\\n[WhatsApp ...] Someone: hello\\n\\n[Current message - respond to this]\\n[WhatsApp ...] Jake: /status\\n[from: Jake McInteer (+6421807830)]`, - RawBody: "/status", - ChatType: "group", - From: "+1222", - To: "+1222", - SessionKey: "agent:main:whatsapp:group:g1", - Provider: "whatsapp", - Surface: "whatsapp", - SenderE164: "+1222", - CommandAuthorized: true, - }; - - const res = await getReplyFromConfig( - groupMessageCtx, - {}, - { - agents: { - defaults: { - model: "anthropic/claude-opus-4-5", - workspace: path.join(home, "openclaw"), - }, - }, - channels: { whatsapp: { allowFrom: ["+1222"] } }, - session: { store: path.join(home, "sessions.json") }, - }, - ); - - const text = Array.isArray(res) ? res[0]?.text : res?.text; - expect(text).toContain("Session: agent:main:whatsapp:group:g1"); - expect(text).toContain("anthropic/claude-opus-4-5"); - expect(runEmbeddedPiAgent).not.toHaveBeenCalled(); - }); - }); - - it("preserves history when RawBody is provided for command parsing", async () => { + it("handles directives, history, and non-default agent session files", async () => { await withTempHome(async (home) => { vi.mocked(runEmbeddedPiAgent).mockResolvedValue({ payloads: [{ text: "ok" }], @@ -238,11 +147,6 @@ describe("RawBody directive parsing", () => { expect(prompt).toContain('"body": "hello"'); expect(prompt).toContain("status please"); expect(prompt).not.toContain("/think:high"); - }); - }); - - it("reuses non-default agent session files without throwing path validation errors", async () => { - await withTempHome(async (home) => { const agentId = "worker1"; const sessionId = "sess-worker-1"; const sessionKey = `agent:${agentId}:telegram:12345`; @@ -259,6 +163,7 @@ describe("RawBody directive parsing", () => { }, }); + vi.mocked(runEmbeddedPiAgent).mockReset(); vi.mocked(runEmbeddedPiAgent).mockResolvedValue({ payloads: [{ text: "ok" }], meta: { @@ -267,7 +172,7 @@ describe("RawBody directive parsing", () => { }, }); - const res = await getReplyFromConfig( + const resWorker = await getReplyFromConfig( { Body: "hello", From: "telegram:12345", @@ -288,8 +193,8 @@ describe("RawBody directive parsing", () => { }, ); - const text = Array.isArray(res) ? res[0]?.text : res?.text; - expect(text).toBe("ok"); + const textWorker = Array.isArray(resWorker) ? resWorker[0]?.text : resWorker?.text; + expect(textWorker).toBe("ok"); expect(runEmbeddedPiAgent).toHaveBeenCalledOnce(); expect(vi.mocked(runEmbeddedPiAgent).mock.calls[0]?.[0]?.sessionFile).toBe(sessionFile); }); diff --git a/src/auto-reply/reply/commands-core.ts b/src/auto-reply/reply/commands-core.ts index c139fd6f646..e3586708488 100644 --- a/src/auto-reply/reply/commands-core.ts +++ b/src/auto-reply/reply/commands-core.ts @@ -1,3 +1,4 @@ +import fs from "node:fs/promises"; import type { CommandHandler, CommandHandlerResult, @@ -5,6 +6,7 @@ import type { } from "./commands-types.js"; import { logVerbose } from "../../globals.js"; import { createInternalHookEvent, triggerInternalHook } from "../../hooks/internal-hooks.js"; +import { getGlobalHookRunner } from "../../plugins/hook-runner-global.js"; import { resolveSendPolicy } from "../../sessions/send-policy.js"; import { shouldHandleTextCommands } from "../commands-registry.js"; import { handleAllowlistCommand } from "./commands-allowlist.js"; @@ -104,6 +106,48 @@ export async function handleCommands(params: HandleCommandsParams): Promise { + try { + const messages: unknown[] = []; + if (sessionFile) { + const content = await fs.readFile(sessionFile, "utf-8"); + for (const line of content.split("\n")) { + if (!line.trim()) { + continue; + } + try { + const entry = JSON.parse(line); + if (entry.type === "message" && entry.message) { + messages.push(entry.message); + } + } catch { + // skip malformed lines + } + } + } else { + logVerbose("before_reset: no session file available, firing hook with empty messages"); + } + await hookRunner.runBeforeReset( + { sessionFile, messages, reason: commandAction }, + { + agentId: params.sessionKey?.split(":")[0] ?? "main", + sessionKey: params.sessionKey, + sessionId: prevEntry?.sessionId, + workspaceDir: params.workspaceDir, + }, + ); + } catch (err: unknown) { + logVerbose(`before_reset hook failed: ${String(err)}`); + } + })(); + } } const allowTextCommands = shouldHandleTextCommands({ diff --git a/src/auto-reply/reply/dispatch-from-config.test.ts b/src/auto-reply/reply/dispatch-from-config.test.ts index 01c96466965..4cc6657d2a2 100644 --- a/src/auto-reply/reply/dispatch-from-config.test.ts +++ b/src/auto-reply/reply/dispatch-from-config.test.ts @@ -64,6 +64,7 @@ function createDispatcher(): ReplyDispatcher { sendFinalReply: vi.fn(() => true), waitForIdle: vi.fn(async () => {}), getQueuedCounts: vi.fn(() => ({ tool: 0, block: 0, final: 0 })), + markComplete: vi.fn(), }; } diff --git a/src/auto-reply/reply/dispatch-from-config.ts b/src/auto-reply/reply/dispatch-from-config.ts index f04aff0a7b5..45bd75040aa 100644 --- a/src/auto-reply/reply/dispatch-from-config.ts +++ b/src/auto-reply/reply/dispatch-from-config.ts @@ -278,7 +278,6 @@ export async function dispatchReplyFromConfig(params: { } else { queuedFinal = dispatcher.sendFinalReply(payload); } - await dispatcher.waitForIdle(); const counts = dispatcher.getQueuedCounts(); counts.final += routedFinalCount; recordProcessed("completed", { reason: "fast_abort" }); @@ -443,8 +442,6 @@ export async function dispatchReplyFromConfig(params: { } } - await dispatcher.waitForIdle(); - const counts = dispatcher.getQueuedCounts(); counts.final += routedFinalCount; recordProcessed("completed"); diff --git a/src/auto-reply/reply/dispatcher-registry.ts b/src/auto-reply/reply/dispatcher-registry.ts new file mode 100644 index 00000000000..0ef42fbf73f --- /dev/null +++ b/src/auto-reply/reply/dispatcher-registry.ts @@ -0,0 +1,58 @@ +/** + * Global registry for tracking active reply dispatchers. + * Used to ensure gateway restart waits for all replies to complete. + */ + +type TrackedDispatcher = { + readonly id: string; + readonly pending: () => number; + readonly waitForIdle: () => Promise; +}; + +const activeDispatchers = new Set(); +let nextId = 0; + +/** + * Register a reply dispatcher for global tracking. + * Returns an unregister function to call when the dispatcher is no longer needed. + */ +export function registerDispatcher(dispatcher: { + readonly pending: () => number; + readonly waitForIdle: () => Promise; +}): { id: string; unregister: () => void } { + const id = `dispatcher-${++nextId}`; + const tracked: TrackedDispatcher = { + id, + pending: dispatcher.pending, + waitForIdle: dispatcher.waitForIdle, + }; + activeDispatchers.add(tracked); + + const unregister = () => { + activeDispatchers.delete(tracked); + }; + + return { id, unregister }; +} + +/** + * Get the total number of pending replies across all dispatchers. + */ +export function getTotalPendingReplies(): number { + let total = 0; + for (const dispatcher of activeDispatchers) { + total += dispatcher.pending(); + } + return total; +} + +/** + * Clear all registered dispatchers (for testing). + * WARNING: Only use this in test cleanup! + */ +export function clearAllDispatchers(): void { + if (!process.env.VITEST && process.env.NODE_ENV !== "test") { + throw new Error("clearAllDispatchers() is only available in test environments"); + } + activeDispatchers.clear(); +} diff --git a/src/auto-reply/reply/get-reply-run.media-only.test.ts b/src/auto-reply/reply/get-reply-run.media-only.test.ts new file mode 100644 index 00000000000..942146189fa --- /dev/null +++ b/src/auto-reply/reply/get-reply-run.media-only.test.ts @@ -0,0 +1,192 @@ +import { beforeEach, describe, expect, it, vi } from "vitest"; +import { runPreparedReply } from "./get-reply-run.js"; + +vi.mock("../../agents/auth-profiles/session-override.js", () => ({ + resolveSessionAuthProfileOverride: vi.fn().mockResolvedValue(undefined), +})); + +vi.mock("../../agents/pi-embedded.js", () => ({ + abortEmbeddedPiRun: vi.fn().mockReturnValue(false), + isEmbeddedPiRunActive: vi.fn().mockReturnValue(false), + isEmbeddedPiRunStreaming: vi.fn().mockReturnValue(false), + resolveEmbeddedSessionLane: vi.fn().mockReturnValue("session:session-key"), +})); + +vi.mock("../../config/sessions.js", () => ({ + resolveGroupSessionKey: vi.fn().mockReturnValue(undefined), + resolveSessionFilePath: vi.fn().mockReturnValue("/tmp/session.jsonl"), + resolveSessionFilePathOptions: vi.fn().mockReturnValue({}), + updateSessionStore: vi.fn(), +})); + +vi.mock("../../globals.js", () => ({ + logVerbose: vi.fn(), +})); + +vi.mock("../../process/command-queue.js", () => ({ + clearCommandLane: vi.fn().mockReturnValue(0), + getQueueSize: vi.fn().mockReturnValue(0), +})); + +vi.mock("../../routing/session-key.js", () => ({ + normalizeMainKey: vi.fn().mockReturnValue("main"), +})); + +vi.mock("../../utils/provider-utils.js", () => ({ + isReasoningTagProvider: vi.fn().mockReturnValue(false), +})); + +vi.mock("../command-detection.js", () => ({ + hasControlCommand: vi.fn().mockReturnValue(false), +})); + +vi.mock("./agent-runner.js", () => ({ + runReplyAgent: vi.fn().mockResolvedValue({ text: "ok" }), +})); + +vi.mock("./body.js", () => ({ + applySessionHints: vi.fn().mockImplementation(async ({ baseBody }) => baseBody), +})); + +vi.mock("./groups.js", () => ({ + buildGroupIntro: vi.fn().mockReturnValue(""), +})); + +vi.mock("./inbound-meta.js", () => ({ + buildInboundMetaSystemPrompt: vi.fn().mockReturnValue(""), + buildInboundUserContextPrefix: vi.fn().mockReturnValue(""), +})); + +vi.mock("./queue.js", () => ({ + resolveQueueSettings: vi.fn().mockReturnValue({ mode: "followup" }), +})); + +vi.mock("./route-reply.js", () => ({ + routeReply: vi.fn(), +})); + +vi.mock("./session-updates.js", () => ({ + ensureSkillSnapshot: vi.fn().mockImplementation(async ({ sessionEntry, systemSent }) => ({ + sessionEntry, + systemSent, + skillsSnapshot: undefined, + })), + prependSystemEvents: vi.fn().mockImplementation(async ({ prefixedBodyBase }) => prefixedBodyBase), +})); + +vi.mock("./typing-mode.js", () => ({ + resolveTypingMode: vi.fn().mockReturnValue("off"), +})); + +import { runReplyAgent } from "./agent-runner.js"; + +function baseParams( + overrides: Partial[0]> = {}, +): Parameters[0] { + return { + ctx: { + Body: "", + RawBody: "", + CommandBody: "", + ThreadHistoryBody: "Earlier message in this thread", + OriginatingChannel: "slack", + OriginatingTo: "C123", + ChatType: "group", + }, + sessionCtx: { + Body: "", + BodyStripped: "", + ThreadHistoryBody: "Earlier message in this thread", + MediaPath: "/tmp/input.png", + Provider: "slack", + ChatType: "group", + OriginatingChannel: "slack", + OriginatingTo: "C123", + }, + cfg: { session: {}, channels: {}, agents: { defaults: {} } }, + agentId: "default", + agentDir: "/tmp/agent", + agentCfg: {}, + sessionCfg: {}, + commandAuthorized: true, + command: { + isAuthorizedSender: true, + abortKey: "session-key", + ownerList: [], + senderIsOwner: false, + } as never, + commandSource: "", + allowTextCommands: true, + directives: { + hasThinkDirective: false, + thinkLevel: undefined, + } as never, + defaultActivation: "always", + resolvedThinkLevel: "high", + resolvedVerboseLevel: "off", + resolvedReasoningLevel: "off", + resolvedElevatedLevel: "off", + elevatedEnabled: false, + elevatedAllowed: false, + blockStreamingEnabled: false, + resolvedBlockStreamingBreak: "message_end", + modelState: { + resolveDefaultThinkingLevel: async () => "medium", + } as never, + provider: "anthropic", + model: "claude-opus-4-1", + typing: { + onReplyStart: vi.fn().mockResolvedValue(undefined), + cleanup: vi.fn(), + } as never, + defaultProvider: "anthropic", + defaultModel: "claude-opus-4-1", + timeoutMs: 30_000, + isNewSession: true, + resetTriggered: false, + systemSent: true, + sessionKey: "session-key", + workspaceDir: "/tmp/workspace", + abortedLastRun: false, + ...overrides, + }; +} + +describe("runPreparedReply media-only handling", () => { + beforeEach(() => { + vi.clearAllMocks(); + }); + + it("allows media-only prompts and preserves thread context in queued followups", async () => { + const result = await runPreparedReply(baseParams()); + expect(result).toEqual({ text: "ok" }); + + const call = vi.mocked(runReplyAgent).mock.calls[0]?.[0]; + expect(call).toBeTruthy(); + expect(call?.followupRun.prompt).toContain("[Thread history - for context]"); + expect(call?.followupRun.prompt).toContain("Earlier message in this thread"); + expect(call?.followupRun.prompt).toContain("[User sent media without caption]"); + }); + + it("returns the empty-body reply when there is no text and no media", async () => { + const result = await runPreparedReply( + baseParams({ + ctx: { + Body: "", + RawBody: "", + CommandBody: "", + }, + sessionCtx: { + Body: "", + BodyStripped: "", + Provider: "slack", + }, + }), + ); + + expect(result).toEqual({ + text: "I didn't receive any text in your message. Please resend or add a caption.", + }); + expect(vi.mocked(runReplyAgent)).not.toHaveBeenCalled(); + }); +}); diff --git a/src/auto-reply/reply/get-reply-run.ts b/src/auto-reply/reply/get-reply-run.ts index 5fc6acd45ff..f85446ecec9 100644 --- a/src/auto-reply/reply/get-reply-run.ts +++ b/src/auto-reply/reply/get-reply-run.ts @@ -221,7 +221,10 @@ export async function runPreparedReply( ? baseBodyFinal : [inboundUserContext, baseBodyFinal].filter(Boolean).join("\n\n"); const baseBodyTrimmed = baseBodyForPrompt.trim(); - if (!baseBodyTrimmed) { + const hasMediaAttachment = Boolean( + sessionCtx.MediaPath || (sessionCtx.MediaPaths && sessionCtx.MediaPaths.length > 0), + ); + if (!baseBodyTrimmed && !hasMediaAttachment) { await typing.onReplyStart(); logVerbose("Inbound body empty after normalization; skipping agent run"); typing.cleanup(); @@ -229,8 +232,13 @@ export async function runPreparedReply( text: "I didn't receive any text in your message. Please resend or add a caption.", }; } + // When the user sends media without text, provide a minimal body so the agent + // run proceeds and the image/document is injected by the embedded runner. + const effectiveBaseBody = baseBodyTrimmed + ? baseBodyForPrompt + : "[User sent media without caption]"; let prefixedBodyBase = await applySessionHints({ - baseBody: baseBodyForPrompt, + baseBody: effectiveBaseBody, abortedLastRun, sessionEntry, sessionStore, @@ -337,7 +345,7 @@ export async function runPreparedReply( sessionEntry, resolveSessionFilePathOptions({ agentId, storePath }), ); - const queueBodyBase = [threadContextNote, baseBodyForPrompt].filter(Boolean).join("\n\n"); + const queueBodyBase = [threadContextNote, effectiveBaseBody].filter(Boolean).join("\n\n"); const queuedBody = mediaNote ? [mediaNote, mediaReplyHint, queueBodyBase].filter(Boolean).join("\n").trim() : queueBodyBase; diff --git a/src/auto-reply/reply/reply-dispatcher.ts b/src/auto-reply/reply/reply-dispatcher.ts index 270efb001e5..9027af0693d 100644 --- a/src/auto-reply/reply/reply-dispatcher.ts +++ b/src/auto-reply/reply/reply-dispatcher.ts @@ -3,6 +3,7 @@ import type { GetReplyOptions, ReplyPayload } from "../types.js"; import type { ResponsePrefixContext } from "./response-prefix-template.js"; import type { TypingController } from "./typing.js"; import { sleep } from "../../utils.js"; +import { registerDispatcher } from "./dispatcher-registry.js"; import { normalizeReplyPayload, type NormalizeReplySkipReason } from "./normalize-reply.js"; export type ReplyDispatchKind = "tool" | "block" | "final"; @@ -74,6 +75,7 @@ export type ReplyDispatcher = { sendFinalReply: (payload: ReplyPayload) => boolean; waitForIdle: () => Promise; getQueuedCounts: () => Record; + markComplete: () => void; }; type NormalizeReplyPayloadInternalOptions = Pick< @@ -101,7 +103,10 @@ function normalizeReplyPayloadInternal( export function createReplyDispatcher(options: ReplyDispatcherOptions): ReplyDispatcher { let sendChain: Promise = Promise.resolve(); // Track in-flight deliveries so we can emit a reliable "idle" signal. - let pending = 0; + // Start with pending=1 as a "reservation" to prevent premature gateway restart. + // This is decremented when markComplete() is called to signal no more replies will come. + let pending = 1; + let completeCalled = false; // Track whether we've sent a block reply (for human delay - skip delay on first block). let sentFirstBlock = false; // Serialize outbound replies to preserve tool/block/final order. @@ -111,6 +116,12 @@ export function createReplyDispatcher(options: ReplyDispatcherOptions): ReplyDis final: 0, }; + // Register this dispatcher globally for gateway restart coordination. + const { unregister } = registerDispatcher({ + pending: () => pending, + waitForIdle: () => sendChain, + }); + const enqueue = (kind: ReplyDispatchKind, payload: ReplyPayload) => { const normalized = normalizeReplyPayloadInternal(payload, { responsePrefix: options.responsePrefix, @@ -140,6 +151,8 @@ export function createReplyDispatcher(options: ReplyDispatcherOptions): ReplyDis await sleep(delayMs); } } + // Safe: deliver is called inside an async .then() callback, so even a synchronous + // throw becomes a rejection that flows through .catch()/.finally(), ensuring cleanup. await options.deliver(normalized, { kind }); }) .catch((err) => { @@ -147,19 +160,49 @@ export function createReplyDispatcher(options: ReplyDispatcherOptions): ReplyDis }) .finally(() => { pending -= 1; + // Clear reservation if: + // 1. pending is now 1 (just the reservation left) + // 2. markComplete has been called + // 3. No more replies will be enqueued + if (pending === 1 && completeCalled) { + pending -= 1; // Clear the reservation + } if (pending === 0) { + // Unregister from global tracking when idle. + unregister(); options.onIdle?.(); } }); return true; }; + const markComplete = () => { + if (completeCalled) { + return; + } + completeCalled = true; + // If no replies were enqueued (pending is still 1 = just the reservation), + // schedule clearing the reservation after current microtasks complete. + // This gives any in-flight enqueue() calls a chance to increment pending. + void Promise.resolve().then(() => { + if (pending === 1 && completeCalled) { + // Still just the reservation, no replies were enqueued + pending -= 1; + if (pending === 0) { + unregister(); + options.onIdle?.(); + } + } + }); + }; + return { sendToolResult: (payload) => enqueue("tool", payload), sendBlockReply: (payload) => enqueue("block", payload), sendFinalReply: (payload) => enqueue("final", payload), waitForIdle: () => sendChain, getQueuedCounts: () => ({ ...queuedCounts }), + markComplete, }; } diff --git a/src/auto-reply/reply/reply-routing.test.ts b/src/auto-reply/reply/reply-routing.test.ts index 6637c6c1401..3d5179d6c0c 100644 --- a/src/auto-reply/reply/reply-routing.test.ts +++ b/src/auto-reply/reply/reply-routing.test.ts @@ -100,6 +100,8 @@ describe("createReplyDispatcher", () => { dispatcher.sendFinalReply({ text: "two" }); await dispatcher.waitForIdle(); + dispatcher.markComplete(); + await Promise.resolve(); expect(onIdle).toHaveBeenCalledTimes(1); }); diff --git a/src/browser/control-service.ts b/src/browser/control-service.ts index 93bb89e93dd..55445fce603 100644 --- a/src/browser/control-service.ts +++ b/src/browser/control-service.ts @@ -3,7 +3,11 @@ import { createSubsystemLogger } from "../logging/subsystem.js"; import { resolveBrowserConfig, resolveProfile } from "./config.js"; import { ensureBrowserControlAuth } from "./control-auth.js"; import { ensureChromeExtensionRelayServer } from "./extension-relay.js"; -import { type BrowserServerState, createBrowserRouteContext } from "./server-context.js"; +import { + type BrowserServerState, + createBrowserRouteContext, + listKnownProfileNames, +} from "./server-context.js"; let state: BrowserServerState | null = null; const log = createSubsystemLogger("browser"); @@ -16,6 +20,7 @@ export function getBrowserControlState(): BrowserServerState | null { export function createBrowserControlContext() { return createBrowserRouteContext({ getState: () => state, + refreshConfigFromDisk: true, }); } @@ -71,10 +76,11 @@ export async function stopBrowserControlService(): Promise { const ctx = createBrowserRouteContext({ getState: () => state, + refreshConfigFromDisk: true, }); try { - for (const name of Object.keys(current.resolved.profiles)) { + for (const name of listKnownProfileNames(current)) { try { await ctx.forProfile(name).stopRunningBrowser(); } catch { diff --git a/src/browser/pw-ai-state.ts b/src/browser/pw-ai-state.ts new file mode 100644 index 00000000000..58ce89f30d9 --- /dev/null +++ b/src/browser/pw-ai-state.ts @@ -0,0 +1,9 @@ +let pwAiLoaded = false; + +export function markPwAiLoaded(): void { + pwAiLoaded = true; +} + +export function isPwAiLoaded(): boolean { + return pwAiLoaded; +} diff --git a/src/browser/pw-ai.ts b/src/browser/pw-ai.ts index 72ba680c43d..6da8b410c83 100644 --- a/src/browser/pw-ai.ts +++ b/src/browser/pw-ai.ts @@ -1,3 +1,7 @@ +import { markPwAiLoaded } from "./pw-ai-state.js"; + +markPwAiLoaded(); + export { type BrowserConsoleMessage, closePageByTargetIdViaPlaywright, diff --git a/src/browser/pw-tools-core.clamps-timeoutms-scrollintoview.test.ts b/src/browser/pw-tools-core.clamps-timeoutms-scrollintoview.test.ts index 4a98144ed9d..55216b79bbd 100644 --- a/src/browser/pw-tools-core.clamps-timeoutms-scrollintoview.test.ts +++ b/src/browser/pw-tools-core.clamps-timeoutms-scrollintoview.test.ts @@ -28,10 +28,7 @@ const sessionMocks = vi.hoisted(() => ({ })); vi.mock("./pw-session.js", () => sessionMocks); - -async function importModule() { - return await import("./pw-tools-core.js"); -} +const mod = await import("./pw-tools-core.js"); describe("pw-tools-core", () => { beforeEach(() => { @@ -53,7 +50,6 @@ describe("pw-tools-core", () => { currentRefLocator = { scrollIntoViewIfNeeded }; currentPage = {}; - const mod = await importModule(); await mod.scrollIntoViewViaPlaywright({ cdpUrl: "http://127.0.0.1:18792", targetId: "T1", @@ -70,7 +66,6 @@ describe("pw-tools-core", () => { currentRefLocator = { scrollIntoViewIfNeeded }; currentPage = {}; - const mod = await importModule(); await expect( mod.scrollIntoViewViaPlaywright({ cdpUrl: "http://127.0.0.1:18792", @@ -86,7 +81,6 @@ describe("pw-tools-core", () => { currentRefLocator = { scrollIntoViewIfNeeded }; currentPage = {}; - const mod = await importModule(); await expect( mod.scrollIntoViewViaPlaywright({ cdpUrl: "http://127.0.0.1:18792", @@ -102,7 +96,6 @@ describe("pw-tools-core", () => { currentRefLocator = { click }; currentPage = {}; - const mod = await importModule(); await expect( mod.clickViaPlaywright({ cdpUrl: "http://127.0.0.1:18792", @@ -118,7 +111,6 @@ describe("pw-tools-core", () => { currentRefLocator = { click }; currentPage = {}; - const mod = await importModule(); await expect( mod.clickViaPlaywright({ cdpUrl: "http://127.0.0.1:18792", @@ -136,7 +128,6 @@ describe("pw-tools-core", () => { currentRefLocator = { click }; currentPage = {}; - const mod = await importModule(); await expect( mod.clickViaPlaywright({ cdpUrl: "http://127.0.0.1:18792", diff --git a/src/browser/pw-tools-core.last-file-chooser-arm-wins.test.ts b/src/browser/pw-tools-core.last-file-chooser-arm-wins.test.ts index a197691ca71..baaf3e1ba85 100644 --- a/src/browser/pw-tools-core.last-file-chooser-arm-wins.test.ts +++ b/src/browser/pw-tools-core.last-file-chooser-arm-wins.test.ts @@ -28,10 +28,7 @@ const sessionMocks = vi.hoisted(() => ({ })); vi.mock("./pw-session.js", () => sessionMocks); - -async function importModule() { - return await import("./pw-tools-core.js"); -} +const mod = await import("./pw-tools-core.js"); describe("pw-tools-core", () => { beforeEach(() => { @@ -75,7 +72,6 @@ describe("pw-tools-core", () => { keyboard: { press: vi.fn(async () => {}) }, }; - const mod = await importModule(); await mod.armFileUploadViaPlaywright({ cdpUrl: "http://127.0.0.1:18792", paths: ["/tmp/1"], @@ -101,7 +97,6 @@ describe("pw-tools-core", () => { waitForEvent, }; - const mod = await importModule(); await mod.armDialogViaPlaywright({ cdpUrl: "http://127.0.0.1:18792", accept: true, @@ -145,7 +140,6 @@ describe("pw-tools-core", () => { getByText: vi.fn(() => ({ first: () => ({ waitFor: vi.fn() }) })), }; - const mod = await importModule(); await mod.waitForViaPlaywright({ cdpUrl: "http://127.0.0.1:18792", selector: "#main", diff --git a/src/browser/pw-tools-core.screenshots-element-selector.test.ts b/src/browser/pw-tools-core.screenshots-element-selector.test.ts index a297f7d512e..96a4a06ea54 100644 --- a/src/browser/pw-tools-core.screenshots-element-selector.test.ts +++ b/src/browser/pw-tools-core.screenshots-element-selector.test.ts @@ -28,10 +28,7 @@ const sessionMocks = vi.hoisted(() => ({ })); vi.mock("./pw-session.js", () => sessionMocks); - -async function importModule() { - return await import("./pw-tools-core.js"); -} +const mod = await import("./pw-tools-core.js"); describe("pw-tools-core", () => { beforeEach(() => { @@ -57,7 +54,6 @@ describe("pw-tools-core", () => { screenshot: vi.fn(async () => Buffer.from("P")), }; - const mod = await importModule(); const res = await mod.takeScreenshotViaPlaywright({ cdpUrl: "http://127.0.0.1:18792", targetId: "T1", @@ -78,7 +74,6 @@ describe("pw-tools-core", () => { screenshot: vi.fn(async () => Buffer.from("P")), }; - const mod = await importModule(); const res = await mod.takeScreenshotViaPlaywright({ cdpUrl: "http://127.0.0.1:18792", targetId: "T1", @@ -99,8 +94,6 @@ describe("pw-tools-core", () => { screenshot: vi.fn(async () => Buffer.from("P")), }; - const mod = await importModule(); - await expect( mod.takeScreenshotViaPlaywright({ cdpUrl: "http://127.0.0.1:18792", @@ -127,7 +120,6 @@ describe("pw-tools-core", () => { keyboard: { press: vi.fn(async () => {}) }, }; - const mod = await importModule(); await mod.armFileUploadViaPlaywright({ cdpUrl: "http://127.0.0.1:18792", targetId: "T1", @@ -151,7 +143,6 @@ describe("pw-tools-core", () => { keyboard: { press }, }; - const mod = await importModule(); await mod.armFileUploadViaPlaywright({ cdpUrl: "http://127.0.0.1:18792", paths: [], diff --git a/src/browser/pw-tools-core.waits-next-download-saves-it.test.ts b/src/browser/pw-tools-core.waits-next-download-saves-it.test.ts index 9ff8d1acab0..59d233e0005 100644 --- a/src/browser/pw-tools-core.waits-next-download-saves-it.test.ts +++ b/src/browser/pw-tools-core.waits-next-download-saves-it.test.ts @@ -33,10 +33,7 @@ const tmpDirMocks = vi.hoisted(() => ({ resolvePreferredOpenClawTmpDir: vi.fn(() => "/tmp/openclaw"), })); vi.mock("../infra/tmp-openclaw-dir.js", () => tmpDirMocks); - -async function importModule() { - return await import("./pw-tools-core.js"); -} +const mod = await import("./pw-tools-core.js"); describe("pw-tools-core", () => { beforeEach(() => { @@ -75,7 +72,6 @@ describe("pw-tools-core", () => { currentPage = { on, off }; - const mod = await importModule(); const targetPath = path.resolve("/tmp/file.bin"); const p = mod.waitForDownloadViaPlaywright({ cdpUrl: "http://127.0.0.1:18792", @@ -113,7 +109,6 @@ describe("pw-tools-core", () => { currentPage = { on, off }; - const mod = await importModule(); const targetPath = path.resolve("/tmp/report.pdf"); const p = mod.downloadViaPlaywright({ cdpUrl: "http://127.0.0.1:18792", @@ -152,7 +147,6 @@ describe("pw-tools-core", () => { tmpDirMocks.resolvePreferredOpenClawTmpDir.mockReturnValue("/tmp/openclaw-preferred"); currentPage = { on, off }; - const mod = await importModule(); const p = mod.waitForDownloadViaPlaywright({ cdpUrl: "http://127.0.0.1:18792", targetId: "T1", @@ -194,7 +188,6 @@ describe("pw-tools-core", () => { text: async () => '{"ok":true,"value":123}', }; - const mod = await importModule(); const p = mod.responseBodyViaPlaywright({ cdpUrl: "http://127.0.0.1:18792", targetId: "T1", @@ -218,7 +211,6 @@ describe("pw-tools-core", () => { currentRefLocator = { scrollIntoViewIfNeeded }; currentPage = {}; - const mod = await importModule(); await mod.scrollIntoViewViaPlaywright({ cdpUrl: "http://127.0.0.1:18792", targetId: "T1", @@ -232,7 +224,6 @@ describe("pw-tools-core", () => { currentRefLocator = { scrollIntoViewIfNeeded: vi.fn(async () => {}) }; currentPage = {}; - const mod = await importModule(); await expect( mod.scrollIntoViewViaPlaywright({ cdpUrl: "http://127.0.0.1:18792", diff --git a/src/browser/screenshot.e2e.test.ts b/src/browser/screenshot.e2e.test.ts index f317376bf15..114243896c6 100644 --- a/src/browser/screenshot.e2e.test.ts +++ b/src/browser/screenshot.e2e.test.ts @@ -1,14 +1,17 @@ -import crypto from "node:crypto"; import sharp from "sharp"; import { describe, expect, it } from "vitest"; import { normalizeBrowserScreenshot } from "./screenshot.js"; describe("browser screenshot normalization", () => { it("shrinks oversized images to <=2000x2000 and <=5MB", async () => { - const width = 2300; - const height = 2300; - const raw = crypto.randomBytes(width * height * 3); - const bigPng = await sharp(raw, { raw: { width, height, channels: 3 } }) + const bigPng = await sharp({ + create: { + width: 2100, + height: 2100, + channels: 3, + background: { r: 12, g: 34, b: 56 }, + }, + }) .png({ compressionLevel: 0 }) .toBuffer(); diff --git a/src/browser/server-context.hot-reload-profiles.test.ts b/src/browser/server-context.hot-reload-profiles.test.ts new file mode 100644 index 00000000000..0ff64c23449 --- /dev/null +++ b/src/browser/server-context.hot-reload-profiles.test.ts @@ -0,0 +1,214 @@ +import { beforeEach, describe, expect, it, vi } from "vitest"; + +let cfgProfiles: Record = {}; + +// Simulate module-level cache behavior +let cachedConfig: ReturnType | null = null; + +function buildConfig() { + return { + browser: { + enabled: true, + color: "#FF4500", + headless: true, + defaultProfile: "openclaw", + profiles: { ...cfgProfiles }, + }, + }; +} + +vi.mock("../config/config.js", async (importOriginal) => { + const actual = await importOriginal(); + return { + ...actual, + createConfigIO: () => ({ + loadConfig: () => { + // Always return fresh config for createConfigIO to simulate fresh disk read + return buildConfig(); + }, + }), + loadConfig: () => { + // simulate stale loadConfig that doesn't see updates unless cache cleared + if (!cachedConfig) { + cachedConfig = buildConfig(); + } + return cachedConfig; + }, + clearConfigCache: vi.fn(() => { + // Clear the simulated cache + cachedConfig = null; + }), + writeConfigFile: vi.fn(async () => {}), + }; +}); + +vi.mock("./chrome.js", () => ({ + isChromeCdpReady: vi.fn(async () => false), + isChromeReachable: vi.fn(async () => false), + launchOpenClawChrome: vi.fn(async () => { + throw new Error("launch disabled"); + }), + resolveOpenClawUserDataDir: vi.fn(() => "/tmp/openclaw"), + stopOpenClawChrome: vi.fn(async () => {}), +})); + +vi.mock("./cdp.js", () => ({ + createTargetViaCdp: vi.fn(async () => { + throw new Error("cdp disabled"); + }), + normalizeCdpWsUrl: vi.fn((wsUrl: string) => wsUrl), + snapshotAria: vi.fn(async () => ({ nodes: [] })), + getHeadersWithAuth: vi.fn(() => ({})), + appendCdpPath: vi.fn((cdpUrl: string, path: string) => `${cdpUrl}${path}`), +})); + +vi.mock("./pw-ai.js", () => ({ + closePlaywrightBrowserConnection: vi.fn(async () => {}), +})); + +vi.mock("../media/store.js", () => ({ + ensureMediaDir: vi.fn(async () => {}), + saveMediaBuffer: vi.fn(async () => ({ path: "/tmp/fake.png" })), +})); + +describe("server-context hot-reload profiles", () => { + beforeEach(() => { + vi.resetModules(); + cfgProfiles = { + openclaw: { cdpPort: 18800, color: "#FF4500" }, + }; + cachedConfig = null; // Clear simulated cache + }); + + it("forProfile hot-reloads newly added profiles from config", async () => { + // Start with only openclaw profile + const { createBrowserRouteContext } = await import("./server-context.js"); + const { resolveBrowserConfig } = await import("./config.js"); + const { loadConfig } = await import("../config/config.js"); + + // 1. Prime the cache by calling loadConfig() first + const cfg = loadConfig(); + const resolved = resolveBrowserConfig(cfg.browser, cfg); + + // Verify cache is primed (without desktop) + expect(cfg.browser.profiles.desktop).toBeUndefined(); + const state = { + server: null, + port: 18791, + resolved, + profiles: new Map(), + }; + + const ctx = createBrowserRouteContext({ + getState: () => state, + refreshConfigFromDisk: true, + }); + + // Initially, "desktop" profile should not exist + expect(() => ctx.forProfile("desktop")).toThrow(/not found/); + + // 2. Simulate adding a new profile to config (like user editing openclaw.json) + cfgProfiles.desktop = { cdpUrl: "http://127.0.0.1:9222", color: "#0066CC" }; + + // 3. Verify without clearConfigCache, loadConfig() still returns stale cached value + const staleCfg = loadConfig(); + expect(staleCfg.browser.profiles.desktop).toBeUndefined(); // Cache is stale! + + // 4. Now forProfile should hot-reload (calls createConfigIO().loadConfig() internally) + // It should NOT clear the global cache + const profileCtx = ctx.forProfile("desktop"); + expect(profileCtx.profile.name).toBe("desktop"); + expect(profileCtx.profile.cdpUrl).toBe("http://127.0.0.1:9222"); + + // 5. Verify the new profile was merged into the cached state + expect(state.resolved.profiles.desktop).toBeDefined(); + + // 6. Verify GLOBAL cache was NOT cleared - subsequent simple loadConfig() still sees STALE value + // This confirms the fix: we read fresh config for the specific profile lookup without flushing the global cache + const stillStaleCfg = loadConfig(); + expect(stillStaleCfg.browser.profiles.desktop).toBeUndefined(); + + // Verify clearConfigCache was not called + const { clearConfigCache } = await import("../config/config.js"); + expect(clearConfigCache).not.toHaveBeenCalled(); + }); + + it("forProfile still throws for profiles that don't exist in fresh config", async () => { + const { createBrowserRouteContext } = await import("./server-context.js"); + const { resolveBrowserConfig } = await import("./config.js"); + const { loadConfig } = await import("../config/config.js"); + + const cfg = loadConfig(); + const resolved = resolveBrowserConfig(cfg.browser, cfg); + const state = { + server: null, + port: 18791, + resolved, + profiles: new Map(), + }; + + const ctx = createBrowserRouteContext({ + getState: () => state, + refreshConfigFromDisk: true, + }); + + // Profile that doesn't exist anywhere should still throw + expect(() => ctx.forProfile("nonexistent")).toThrow(/not found/); + }); + + it("forProfile refreshes existing profile config after loadConfig cache updates", async () => { + const { createBrowserRouteContext } = await import("./server-context.js"); + const { resolveBrowserConfig } = await import("./config.js"); + const { loadConfig } = await import("../config/config.js"); + + const cfg = loadConfig(); + const resolved = resolveBrowserConfig(cfg.browser, cfg); + const state = { + server: null, + port: 18791, + resolved, + profiles: new Map(), + }; + + const ctx = createBrowserRouteContext({ + getState: () => state, + refreshConfigFromDisk: true, + }); + + const before = ctx.forProfile("openclaw"); + expect(before.profile.cdpPort).toBe(18800); + + cfgProfiles.openclaw = { cdpPort: 19999, color: "#FF4500" }; + cachedConfig = null; + + const after = ctx.forProfile("openclaw"); + expect(after.profile.cdpPort).toBe(19999); + expect(state.resolved.profiles.openclaw?.cdpPort).toBe(19999); + }); + + it("listProfiles refreshes config before enumerating profiles", async () => { + const { createBrowserRouteContext } = await import("./server-context.js"); + const { resolveBrowserConfig } = await import("./config.js"); + const { loadConfig } = await import("../config/config.js"); + + const cfg = loadConfig(); + const resolved = resolveBrowserConfig(cfg.browser, cfg); + const state = { + server: null, + port: 18791, + resolved, + profiles: new Map(), + }; + + const ctx = createBrowserRouteContext({ + getState: () => state, + refreshConfigFromDisk: true, + }); + + cfgProfiles.desktop = { cdpPort: 19999, color: "#0066CC" }; + cachedConfig = null; + + const profiles = await ctx.listProfiles(); + expect(profiles.some((p) => p.name === "desktop")).toBe(true); + }); +}); diff --git a/src/browser/server-context.list-known-profile-names.test.ts b/src/browser/server-context.list-known-profile-names.test.ts new file mode 100644 index 00000000000..04c897563e9 --- /dev/null +++ b/src/browser/server-context.list-known-profile-names.test.ts @@ -0,0 +1,40 @@ +import { describe, expect, it } from "vitest"; +import type { BrowserServerState } from "./server-context.js"; +import { resolveBrowserConfig, resolveProfile } from "./config.js"; +import { listKnownProfileNames } from "./server-context.js"; + +describe("browser server-context listKnownProfileNames", () => { + it("includes configured and runtime-only profile names", () => { + const resolved = resolveBrowserConfig({ + defaultProfile: "openclaw", + profiles: { + openclaw: { cdpPort: 18800, color: "#FF4500" }, + }, + }); + const openclaw = resolveProfile(resolved, "openclaw"); + if (!openclaw) { + throw new Error("expected openclaw profile"); + } + + const state: BrowserServerState = { + server: null as unknown as BrowserServerState["server"], + port: 18791, + resolved, + profiles: new Map([ + [ + "stale-removed", + { + profile: { ...openclaw, name: "stale-removed" }, + running: null, + }, + ], + ]), + }; + + expect(listKnownProfileNames(state).toSorted()).toEqual([ + "chrome", + "openclaw", + "stale-removed", + ]); + }); +}); diff --git a/src/browser/server-context.ts b/src/browser/server-context.ts index d6e0e8f0474..658e75b3db1 100644 --- a/src/browser/server-context.ts +++ b/src/browser/server-context.ts @@ -2,6 +2,7 @@ import fs from "node:fs"; import type { ResolvedBrowserProfile } from "./config.js"; import type { PwAiModule } from "./pw-ai-module.js"; import type { + BrowserServerState, BrowserRouteContext, BrowserTab, ContextOptions, @@ -9,6 +10,7 @@ import type { ProfileRuntimeState, ProfileStatus, } from "./server-context.types.js"; +import { createConfigIO, loadConfig } from "../config/config.js"; import { appendCdpPath, createTargetViaCdp, getHeadersWithAuth, normalizeCdpWsUrl } from "./cdp.js"; import { isChromeCdpReady, @@ -17,7 +19,7 @@ import { resolveOpenClawUserDataDir, stopOpenClawChrome, } from "./chrome.js"; -import { resolveProfile } from "./config.js"; +import { resolveBrowserConfig, resolveProfile } from "./config.js"; import { ensureChromeExtensionRelayServer, stopChromeExtensionRelayServer, @@ -35,6 +37,14 @@ export type { ProfileStatus, } from "./server-context.types.js"; +export function listKnownProfileNames(state: BrowserServerState): string[] { + const names = new Set(Object.keys(state.resolved.profiles)); + for (const name of state.profiles.keys()) { + names.add(name); + } + return [...names]; +} + /** * Normalize a CDP WebSocket URL to use the correct base URL. */ @@ -559,6 +569,8 @@ function createProfileContext( } export function createBrowserRouteContext(opts: ContextOptions): BrowserRouteContext { + const refreshConfigFromDisk = opts.refreshConfigFromDisk === true; + const state = () => { const current = opts.getState(); if (!current) { @@ -567,10 +579,53 @@ export function createBrowserRouteContext(opts: ContextOptions): BrowserRouteCon return current; }; + const applyResolvedConfig = ( + current: BrowserServerState, + freshResolved: BrowserServerState["resolved"], + ) => { + current.resolved = freshResolved; + for (const [name, runtime] of current.profiles) { + const nextProfile = resolveProfile(freshResolved, name); + if (nextProfile) { + runtime.profile = nextProfile; + continue; + } + if (!runtime.running) { + current.profiles.delete(name); + } + } + }; + + const refreshResolvedConfig = (current: BrowserServerState) => { + if (!refreshConfigFromDisk) { + return; + } + const cfg = loadConfig(); + const freshResolved = resolveBrowserConfig(cfg.browser, cfg); + applyResolvedConfig(current, freshResolved); + }; + + const refreshResolvedConfigFresh = (current: BrowserServerState) => { + if (!refreshConfigFromDisk) { + return; + } + const freshCfg = createConfigIO().loadConfig(); + const freshResolved = resolveBrowserConfig(freshCfg.browser, freshCfg); + applyResolvedConfig(current, freshResolved); + }; + const forProfile = (profileName?: string): ProfileContext => { const current = state(); + refreshResolvedConfig(current); const name = profileName ?? current.resolved.defaultProfile; - const profile = resolveProfile(current.resolved, name); + let profile = resolveProfile(current.resolved, name); + + // Hot-reload: try fresh config if profile not found + if (!profile) { + refreshResolvedConfigFresh(current); + profile = resolveProfile(current.resolved, name); + } + if (!profile) { const available = Object.keys(current.resolved.profiles).join(", "); throw new Error(`Profile "${name}" not found. Available profiles: ${available || "(none)"}`); @@ -580,6 +635,7 @@ export function createBrowserRouteContext(opts: ContextOptions): BrowserRouteCon const listProfiles = async (): Promise => { const current = state(); + refreshResolvedConfig(current); const result: ProfileStatus[] = []; for (const name of Object.keys(current.resolved.profiles)) { diff --git a/src/browser/server-context.types.ts b/src/browser/server-context.types.ts index 62a8ae02862..d9360b84916 100644 --- a/src/browser/server-context.types.ts +++ b/src/browser/server-context.types.ts @@ -72,4 +72,5 @@ export type ProfileStatus = { export type ContextOptions = { getState: () => BrowserServerState | null; onEnsureAttachTarget?: (profile: ResolvedBrowserProfile) => Promise; + refreshConfigFromDisk?: boolean; }; diff --git a/src/browser/server.agent-contract-form-layout-act-commands.test.ts b/src/browser/server.agent-contract-form-layout-act-commands.test.ts index a63eef29c19..2c5c2234740 100644 --- a/src/browser/server.agent-contract-form-layout-act-commands.test.ts +++ b/src/browser/server.agent-contract-form-layout-act-commands.test.ts @@ -156,6 +156,9 @@ vi.mock("./screenshot.js", () => ({ })), })); +const { startBrowserControlServerFromConfig, stopBrowserControlServer } = + await import("./server.js"); + async function getFreePort(): Promise { while (true) { const port = await new Promise((resolve, reject) => { @@ -274,12 +277,10 @@ describe("browser control server", () => { } else { process.env.OPENCLAW_GATEWAY_PORT = prevGatewayPort; } - const { stopBrowserControlServer } = await import("./server.js"); await stopBrowserControlServer(); }); const startServerAndBase = async () => { - const { startBrowserControlServerFromConfig } = await import("./server.js"); await startBrowserControlServerFromConfig(); const base = `http://127.0.0.1:${testPort}`; await realFetch(`${base}/start`, { method: "POST" }).then((r) => r.json()); diff --git a/src/browser/server.agent-contract-snapshot-endpoints.test.ts b/src/browser/server.agent-contract-snapshot-endpoints.test.ts index ab8c70317d2..8c4530a91a2 100644 --- a/src/browser/server.agent-contract-snapshot-endpoints.test.ts +++ b/src/browser/server.agent-contract-snapshot-endpoints.test.ts @@ -154,6 +154,9 @@ vi.mock("./screenshot.js", () => ({ })), })); +const { startBrowserControlServerFromConfig, stopBrowserControlServer } = + await import("./server.js"); + async function getFreePort(): Promise { while (true) { const port = await new Promise((resolve, reject) => { @@ -271,12 +274,10 @@ describe("browser control server", () => { } else { process.env.OPENCLAW_GATEWAY_PORT = prevGatewayPort; } - const { stopBrowserControlServer } = await import("./server.js"); await stopBrowserControlServer(); }); const startServerAndBase = async () => { - const { startBrowserControlServerFromConfig } = await import("./server.js"); await startBrowserControlServerFromConfig(); const base = `http://127.0.0.1:${testPort}`; await realFetch(`${base}/start`, { method: "POST" }).then((r) => r.json()); diff --git a/src/browser/server.covers-additional-endpoint-branches.test.ts b/src/browser/server.covers-additional-endpoint-branches.test.ts deleted file mode 100644 index 70fa7bfefb3..00000000000 --- a/src/browser/server.covers-additional-endpoint-branches.test.ts +++ /dev/null @@ -1,511 +0,0 @@ -import { type AddressInfo, createServer } from "node:net"; -import { fetch as realFetch } from "undici"; -import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; - -let testPort = 0; -let _cdpBaseUrl = ""; -let reachable = false; -let cfgAttachOnly = false; -let createTargetId: string | null = null; -let prevGatewayPort: string | undefined; - -const cdpMocks = vi.hoisted(() => ({ - createTargetViaCdp: vi.fn(async () => { - throw new Error("cdp disabled"); - }), - snapshotAria: vi.fn(async () => ({ - nodes: [{ ref: "1", role: "link", name: "x", depth: 0 }], - })), -})); - -const pwMocks = vi.hoisted(() => ({ - armDialogViaPlaywright: vi.fn(async () => {}), - armFileUploadViaPlaywright: vi.fn(async () => {}), - clickViaPlaywright: vi.fn(async () => {}), - closePageViaPlaywright: vi.fn(async () => {}), - closePlaywrightBrowserConnection: vi.fn(async () => {}), - downloadViaPlaywright: vi.fn(async () => ({ - url: "https://example.com/report.pdf", - suggestedFilename: "report.pdf", - path: "/tmp/report.pdf", - })), - dragViaPlaywright: vi.fn(async () => {}), - evaluateViaPlaywright: vi.fn(async () => "ok"), - fillFormViaPlaywright: vi.fn(async () => {}), - getConsoleMessagesViaPlaywright: vi.fn(async () => []), - hoverViaPlaywright: vi.fn(async () => {}), - scrollIntoViewViaPlaywright: vi.fn(async () => {}), - navigateViaPlaywright: vi.fn(async () => ({ url: "https://example.com" })), - pdfViaPlaywright: vi.fn(async () => ({ buffer: Buffer.from("pdf") })), - pressKeyViaPlaywright: vi.fn(async () => {}), - responseBodyViaPlaywright: vi.fn(async () => ({ - url: "https://example.com/api/data", - status: 200, - headers: { "content-type": "application/json" }, - body: '{"ok":true}', - })), - resizeViewportViaPlaywright: vi.fn(async () => {}), - selectOptionViaPlaywright: vi.fn(async () => {}), - setInputFilesViaPlaywright: vi.fn(async () => {}), - snapshotAiViaPlaywright: vi.fn(async () => ({ snapshot: "ok" })), - takeScreenshotViaPlaywright: vi.fn(async () => ({ - buffer: Buffer.from("png"), - })), - typeViaPlaywright: vi.fn(async () => {}), - waitForDownloadViaPlaywright: vi.fn(async () => ({ - url: "https://example.com/report.pdf", - suggestedFilename: "report.pdf", - path: "/tmp/report.pdf", - })), - waitForViaPlaywright: vi.fn(async () => {}), -})); - -function makeProc(pid = 123) { - const handlers = new Map void>>(); - return { - pid, - killed: false, - exitCode: null as number | null, - on: (event: string, cb: (...args: unknown[]) => void) => { - handlers.set(event, [...(handlers.get(event) ?? []), cb]); - return undefined; - }, - emitExit: () => { - for (const cb of handlers.get("exit") ?? []) { - cb(0); - } - }, - kill: () => { - return true; - }, - }; -} - -const proc = makeProc(); - -vi.mock("../config/config.js", async (importOriginal) => { - const actual = await importOriginal(); - return { - ...actual, - loadConfig: () => ({ - browser: { - enabled: true, - color: "#FF4500", - attachOnly: cfgAttachOnly, - headless: true, - defaultProfile: "openclaw", - profiles: { - openclaw: { cdpPort: testPort + 1, color: "#FF4500" }, - }, - }, - }), - writeConfigFile: vi.fn(async () => {}), - }; -}); - -const launchCalls = vi.hoisted(() => [] as Array<{ port: number }>); -vi.mock("./chrome.js", () => ({ - isChromeCdpReady: vi.fn(async () => reachable), - isChromeReachable: vi.fn(async () => reachable), - launchOpenClawChrome: vi.fn(async (_resolved: unknown, profile: { cdpPort: number }) => { - launchCalls.push({ port: profile.cdpPort }); - reachable = true; - return { - pid: 123, - exe: { kind: "chrome", path: "/fake/chrome" }, - userDataDir: "/tmp/openclaw", - cdpPort: profile.cdpPort, - startedAt: Date.now(), - proc, - }; - }), - resolveOpenClawUserDataDir: vi.fn(() => "/tmp/openclaw"), - stopOpenClawChrome: vi.fn(async () => { - reachable = false; - }), -})); - -vi.mock("./cdp.js", () => ({ - createTargetViaCdp: cdpMocks.createTargetViaCdp, - normalizeCdpWsUrl: vi.fn((wsUrl: string) => wsUrl), - snapshotAria: cdpMocks.snapshotAria, - getHeadersWithAuth: vi.fn(() => ({})), - appendCdpPath: vi.fn((cdpUrl: string, path: string) => { - const base = cdpUrl.replace(/\/$/, ""); - const suffix = path.startsWith("/") ? path : `/${path}`; - return `${base}${suffix}`; - }), -})); - -vi.mock("./pw-ai.js", () => pwMocks); - -vi.mock("../media/store.js", () => ({ - ensureMediaDir: vi.fn(async () => {}), - saveMediaBuffer: vi.fn(async () => ({ path: "/tmp/fake.png" })), -})); - -vi.mock("./screenshot.js", () => ({ - DEFAULT_BROWSER_SCREENSHOT_MAX_BYTES: 128, - DEFAULT_BROWSER_SCREENSHOT_MAX_SIDE: 64, - normalizeBrowserScreenshot: vi.fn(async (buf: Buffer) => ({ - buffer: buf, - contentType: "image/png", - })), -})); - -async function getFreePort(): Promise { - while (true) { - const port = await new Promise((resolve, reject) => { - const s = createServer(); - s.once("error", reject); - s.listen(0, "127.0.0.1", () => { - const assigned = (s.address() as AddressInfo).port; - s.close((err) => (err ? reject(err) : resolve(assigned))); - }); - }); - if (port < 65535) { - return port; - } - } -} - -function makeResponse( - body: unknown, - init?: { ok?: boolean; status?: number; text?: string }, -): Response { - const ok = init?.ok ?? true; - const status = init?.status ?? 200; - const text = init?.text ?? ""; - return { - ok, - status, - json: async () => body, - text: async () => text, - } as unknown as Response; -} - -describe("browser control server", () => { - beforeEach(async () => { - reachable = false; - cfgAttachOnly = false; - createTargetId = null; - - cdpMocks.createTargetViaCdp.mockImplementation(async () => { - if (createTargetId) { - return { targetId: createTargetId }; - } - throw new Error("cdp disabled"); - }); - - for (const fn of Object.values(pwMocks)) { - fn.mockClear(); - } - for (const fn of Object.values(cdpMocks)) { - fn.mockClear(); - } - - testPort = await getFreePort(); - _cdpBaseUrl = `http://127.0.0.1:${testPort + 1}`; - prevGatewayPort = process.env.OPENCLAW_GATEWAY_PORT; - process.env.OPENCLAW_GATEWAY_PORT = String(testPort - 2); - - // Minimal CDP JSON endpoints used by the server. - let putNewCalls = 0; - vi.stubGlobal( - "fetch", - vi.fn(async (url: string, init?: RequestInit) => { - const u = String(url); - if (u.includes("/json/list")) { - if (!reachable) { - return makeResponse([]); - } - return makeResponse([ - { - id: "abcd1234", - title: "Tab", - url: "https://example.com", - webSocketDebuggerUrl: "ws://127.0.0.1/devtools/page/abcd1234", - type: "page", - }, - { - id: "abce9999", - title: "Other", - url: "https://other", - webSocketDebuggerUrl: "ws://127.0.0.1/devtools/page/abce9999", - type: "page", - }, - ]); - } - if (u.includes("/json/new?")) { - if (init?.method === "PUT") { - putNewCalls += 1; - if (putNewCalls === 1) { - return makeResponse({}, { ok: false, status: 405, text: "" }); - } - } - return makeResponse({ - id: "newtab1", - title: "", - url: "about:blank", - webSocketDebuggerUrl: "ws://127.0.0.1/devtools/page/newtab1", - type: "page", - }); - } - if (u.includes("/json/activate/")) { - return makeResponse("ok"); - } - if (u.includes("/json/close/")) { - return makeResponse("ok"); - } - return makeResponse({}, { ok: false, status: 500, text: "unexpected" }); - }), - ); - }); - - afterEach(async () => { - vi.unstubAllGlobals(); - vi.restoreAllMocks(); - if (prevGatewayPort === undefined) { - delete process.env.OPENCLAW_GATEWAY_PORT; - } else { - process.env.OPENCLAW_GATEWAY_PORT = prevGatewayPort; - } - const { stopBrowserControlServer } = await import("./server.js"); - await stopBrowserControlServer(); - }); - - it("covers additional endpoint branches", async () => { - const { startBrowserControlServerFromConfig } = await import("./server.js"); - await startBrowserControlServerFromConfig(); - const base = `http://127.0.0.1:${testPort}`; - - const tabsWhenStopped = (await realFetch(`${base}/tabs`).then((r) => r.json())) as { - running: boolean; - tabs: unknown[]; - }; - expect(tabsWhenStopped.running).toBe(false); - expect(Array.isArray(tabsWhenStopped.tabs)).toBe(true); - - const focusStopped = await realFetch(`${base}/tabs/focus`, { - method: "POST", - headers: { "Content-Type": "application/json" }, - body: JSON.stringify({ targetId: "abcd" }), - }); - expect(focusStopped.status).toBe(409); - - await realFetch(`${base}/start`, { method: "POST" }).then((r) => r.json()); - - const focusMissing = await realFetch(`${base}/tabs/focus`, { - method: "POST", - headers: { "Content-Type": "application/json" }, - body: JSON.stringify({ targetId: "zzz" }), - }); - expect(focusMissing.status).toBe(404); - - const delAmbiguous = await realFetch(`${base}/tabs/abc`, { - method: "DELETE", - }); - expect(delAmbiguous.status).toBe(409); - - const snapAmbiguous = await realFetch(`${base}/snapshot?format=aria&targetId=abc`); - expect(snapAmbiguous.status).toBe(409); - }); -}); - -describe("backward compatibility (profile parameter)", () => { - beforeEach(async () => { - reachable = false; - cfgAttachOnly = false; - createTargetId = null; - - for (const fn of Object.values(pwMocks)) { - fn.mockClear(); - } - for (const fn of Object.values(cdpMocks)) { - fn.mockClear(); - } - - testPort = await getFreePort(); - _cdpBaseUrl = `http://127.0.0.1:${testPort + 1}`; - prevGatewayPort = process.env.OPENCLAW_GATEWAY_PORT; - process.env.OPENCLAW_GATEWAY_PORT = String(testPort - 2); - - prevGatewayPort = process.env.OPENCLAW_GATEWAY_PORT; - process.env.OPENCLAW_GATEWAY_PORT = String(testPort - 2); - - vi.stubGlobal( - "fetch", - vi.fn(async (url: string) => { - const u = String(url); - if (u.includes("/json/list")) { - if (!reachable) { - return makeResponse([]); - } - return makeResponse([ - { - id: "abcd1234", - title: "Tab", - url: "https://example.com", - webSocketDebuggerUrl: "ws://127.0.0.1/devtools/page/abcd1234", - type: "page", - }, - ]); - } - if (u.includes("/json/new?")) { - return makeResponse({ - id: "newtab1", - title: "", - url: "about:blank", - webSocketDebuggerUrl: "ws://127.0.0.1/devtools/page/newtab1", - type: "page", - }); - } - if (u.includes("/json/activate/")) { - return makeResponse("ok"); - } - if (u.includes("/json/close/")) { - return makeResponse("ok"); - } - return makeResponse({}, { ok: false, status: 500, text: "unexpected" }); - }), - ); - }); - - afterEach(async () => { - vi.unstubAllGlobals(); - vi.restoreAllMocks(); - if (prevGatewayPort === undefined) { - delete process.env.OPENCLAW_GATEWAY_PORT; - } else { - process.env.OPENCLAW_GATEWAY_PORT = prevGatewayPort; - } - const { stopBrowserControlServer } = await import("./server.js"); - await stopBrowserControlServer(); - }); - - it("GET / without profile uses default profile", async () => { - const { startBrowserControlServerFromConfig } = await import("./server.js"); - await startBrowserControlServerFromConfig(); - const base = `http://127.0.0.1:${testPort}`; - - const status = (await realFetch(`${base}/`).then((r) => r.json())) as { - running: boolean; - profile?: string; - }; - expect(status.running).toBe(false); - // Should use default profile (openclaw) - expect(status.profile).toBe("openclaw"); - }); - - it("POST /start without profile uses default profile", async () => { - const { startBrowserControlServerFromConfig } = await import("./server.js"); - await startBrowserControlServerFromConfig(); - const base = `http://127.0.0.1:${testPort}`; - - const result = (await realFetch(`${base}/start`, { method: "POST" }).then((r) => r.json())) as { - ok: boolean; - profile?: string; - }; - expect(result.ok).toBe(true); - expect(result.profile).toBe("openclaw"); - }); - - it("POST /stop without profile uses default profile", async () => { - const { startBrowserControlServerFromConfig } = await import("./server.js"); - await startBrowserControlServerFromConfig(); - const base = `http://127.0.0.1:${testPort}`; - - await realFetch(`${base}/start`, { method: "POST" }); - - const result = (await realFetch(`${base}/stop`, { method: "POST" }).then((r) => r.json())) as { - ok: boolean; - profile?: string; - }; - expect(result.ok).toBe(true); - expect(result.profile).toBe("openclaw"); - }); - - it("GET /tabs without profile uses default profile", async () => { - const { startBrowserControlServerFromConfig } = await import("./server.js"); - await startBrowserControlServerFromConfig(); - const base = `http://127.0.0.1:${testPort}`; - - await realFetch(`${base}/start`, { method: "POST" }); - - const result = (await realFetch(`${base}/tabs`).then((r) => r.json())) as { - running: boolean; - tabs: unknown[]; - }; - expect(result.running).toBe(true); - expect(Array.isArray(result.tabs)).toBe(true); - }); - - it("POST /tabs/open without profile uses default profile", async () => { - const { startBrowserControlServerFromConfig } = await import("./server.js"); - await startBrowserControlServerFromConfig(); - const base = `http://127.0.0.1:${testPort}`; - - await realFetch(`${base}/start`, { method: "POST" }); - - const result = (await realFetch(`${base}/tabs/open`, { - method: "POST", - headers: { "Content-Type": "application/json" }, - body: JSON.stringify({ url: "https://example.com" }), - }).then((r) => r.json())) as { targetId?: string }; - expect(result.targetId).toBe("newtab1"); - }); - - it("GET /profiles returns list of profiles", async () => { - const { startBrowserControlServerFromConfig } = await import("./server.js"); - await startBrowserControlServerFromConfig(); - const base = `http://127.0.0.1:${testPort}`; - - const result = (await realFetch(`${base}/profiles`).then((r) => r.json())) as { - profiles: Array<{ name: string }>; - }; - expect(Array.isArray(result.profiles)).toBe(true); - // Should at least have the default openclaw profile - expect(result.profiles.some((p) => p.name === "openclaw")).toBe(true); - }); - - it("GET /tabs?profile=openclaw returns tabs for specified profile", async () => { - const { startBrowserControlServerFromConfig } = await import("./server.js"); - await startBrowserControlServerFromConfig(); - const base = `http://127.0.0.1:${testPort}`; - - await realFetch(`${base}/start`, { method: "POST" }); - - const result = (await realFetch(`${base}/tabs?profile=openclaw`).then((r) => r.json())) as { - running: boolean; - tabs: unknown[]; - }; - expect(result.running).toBe(true); - expect(Array.isArray(result.tabs)).toBe(true); - }); - - it("POST /tabs/open?profile=openclaw opens tab in specified profile", async () => { - const { startBrowserControlServerFromConfig } = await import("./server.js"); - await startBrowserControlServerFromConfig(); - const base = `http://127.0.0.1:${testPort}`; - - await realFetch(`${base}/start`, { method: "POST" }); - - const result = (await realFetch(`${base}/tabs/open?profile=openclaw`, { - method: "POST", - headers: { "Content-Type": "application/json" }, - body: JSON.stringify({ url: "https://example.com" }), - }).then((r) => r.json())) as { targetId?: string }; - expect(result.targetId).toBe("newtab1"); - }); - - it("GET /tabs?profile=unknown returns 404", async () => { - const { startBrowserControlServerFromConfig } = await import("./server.js"); - await startBrowserControlServerFromConfig(); - const base = `http://127.0.0.1:${testPort}`; - - const result = await realFetch(`${base}/tabs?profile=unknown`); - expect(result.status).toBe(404); - const body = (await result.json()) as { error: string }; - expect(body.error).toContain("not found"); - }); -}); diff --git a/src/browser/server.evaluate-disabled-does-not-block-storage.test.ts b/src/browser/server.evaluate-disabled-does-not-block-storage.test.ts index b24438f2787..c7d3f6c9523 100644 --- a/src/browser/server.evaluate-disabled-does-not-block-storage.test.ts +++ b/src/browser/server.evaluate-disabled-does-not-block-storage.test.ts @@ -63,6 +63,9 @@ vi.mock("./server-context.js", async (importOriginal) => { }; }); +const { startBrowserControlServerFromConfig, stopBrowserControlServer } = + await import("./server.js"); + async function getFreePort(): Promise { const probe = createServer(); await new Promise((resolve, reject) => { @@ -95,12 +98,10 @@ describe("browser control evaluate gating", () => { process.env.OPENCLAW_GATEWAY_PORT = prevGatewayPort; } - const { stopBrowserControlServer } = await import("./server.js"); await stopBrowserControlServer(); }); it("blocks act:evaluate but still allows cookies/storage reads", async () => { - const { startBrowserControlServerFromConfig } = await import("./server.js"); await startBrowserControlServerFromConfig(); const base = `http://127.0.0.1:${testPort}`; diff --git a/src/browser/server.post-tabs-open-profile-unknown-returns-404.test.ts b/src/browser/server.post-tabs-open-profile-unknown-returns-404.test.ts index e2c75a85f0e..9ed16d6f3b1 100644 --- a/src/browser/server.post-tabs-open-profile-unknown-returns-404.test.ts +++ b/src/browser/server.post-tabs-open-profile-unknown-returns-404.test.ts @@ -153,6 +153,9 @@ vi.mock("./screenshot.js", () => ({ })), })); +const { startBrowserControlServerFromConfig, stopBrowserControlServer } = + await import("./server.js"); + async function getFreePort(): Promise { while (true) { const port = await new Promise((resolve, reject) => { @@ -270,12 +273,10 @@ describe("browser control server", () => { } else { process.env.OPENCLAW_GATEWAY_PORT = prevGatewayPort; } - const { stopBrowserControlServer } = await import("./server.js"); await stopBrowserControlServer(); }); it("POST /tabs/open?profile=unknown returns 404", async () => { - const { startBrowserControlServerFromConfig } = await import("./server.js"); await startBrowserControlServerFromConfig(); const base = `http://127.0.0.1:${testPort}`; @@ -307,9 +308,6 @@ describe("profile CRUD endpoints", () => { prevGatewayPort = process.env.OPENCLAW_GATEWAY_PORT; process.env.OPENCLAW_GATEWAY_PORT = String(testPort - 2); - prevGatewayPort = process.env.OPENCLAW_GATEWAY_PORT; - process.env.OPENCLAW_GATEWAY_PORT = String(testPort - 2); - vi.stubGlobal( "fetch", vi.fn(async (url: string) => { @@ -330,129 +328,83 @@ describe("profile CRUD endpoints", () => { } else { process.env.OPENCLAW_GATEWAY_PORT = prevGatewayPort; } - const { stopBrowserControlServer } = await import("./server.js"); await stopBrowserControlServer(); }); - it("POST /profiles/create returns 400 for missing name", async () => { - const { startBrowserControlServerFromConfig } = await import("./server.js"); + it("validates profile create/delete endpoints", async () => { await startBrowserControlServerFromConfig(); const base = `http://127.0.0.1:${testPort}`; - const result = await realFetch(`${base}/profiles/create`, { + const createMissingName = await realFetch(`${base}/profiles/create`, { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({}), }); - expect(result.status).toBe(400); - const body = (await result.json()) as { error: string }; - expect(body.error).toContain("name is required"); - }); + expect(createMissingName.status).toBe(400); + const createMissingNameBody = (await createMissingName.json()) as { error: string }; + expect(createMissingNameBody.error).toContain("name is required"); - it("POST /profiles/create returns 400 for invalid name format", async () => { - const { startBrowserControlServerFromConfig } = await import("./server.js"); - await startBrowserControlServerFromConfig(); - const base = `http://127.0.0.1:${testPort}`; - - const result = await realFetch(`${base}/profiles/create`, { + const createInvalidName = await realFetch(`${base}/profiles/create`, { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ name: "Invalid Name!" }), }); - expect(result.status).toBe(400); - const body = (await result.json()) as { error: string }; - expect(body.error).toContain("invalid profile name"); - }); + expect(createInvalidName.status).toBe(400); + const createInvalidNameBody = (await createInvalidName.json()) as { error: string }; + expect(createInvalidNameBody.error).toContain("invalid profile name"); - it("POST /profiles/create returns 409 for duplicate name", async () => { - const { startBrowserControlServerFromConfig } = await import("./server.js"); - await startBrowserControlServerFromConfig(); - const base = `http://127.0.0.1:${testPort}`; - - // "openclaw" already exists as the default profile - const result = await realFetch(`${base}/profiles/create`, { + const createDuplicate = await realFetch(`${base}/profiles/create`, { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ name: "openclaw" }), }); - expect(result.status).toBe(409); - const body = (await result.json()) as { error: string }; - expect(body.error).toContain("already exists"); - }); + expect(createDuplicate.status).toBe(409); + const createDuplicateBody = (await createDuplicate.json()) as { error: string }; + expect(createDuplicateBody.error).toContain("already exists"); - it("POST /profiles/create accepts cdpUrl for remote profiles", async () => { - const { startBrowserControlServerFromConfig } = await import("./server.js"); - await startBrowserControlServerFromConfig(); - const base = `http://127.0.0.1:${testPort}`; - - const result = await realFetch(`${base}/profiles/create`, { + const createRemote = await realFetch(`${base}/profiles/create`, { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ name: "remote", cdpUrl: "http://10.0.0.42:9222" }), }); - expect(result.status).toBe(200); - const body = (await result.json()) as { + expect(createRemote.status).toBe(200); + const createRemoteBody = (await createRemote.json()) as { profile?: string; cdpUrl?: string; isRemote?: boolean; }; - expect(body.profile).toBe("remote"); - expect(body.cdpUrl).toBe("http://10.0.0.42:9222"); - expect(body.isRemote).toBe(true); - }); + expect(createRemoteBody.profile).toBe("remote"); + expect(createRemoteBody.cdpUrl).toBe("http://10.0.0.42:9222"); + expect(createRemoteBody.isRemote).toBe(true); - it("POST /profiles/create returns 400 for invalid cdpUrl", async () => { - const { startBrowserControlServerFromConfig } = await import("./server.js"); - await startBrowserControlServerFromConfig(); - const base = `http://127.0.0.1:${testPort}`; - - const result = await realFetch(`${base}/profiles/create`, { + const createBadRemote = await realFetch(`${base}/profiles/create`, { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ name: "badremote", cdpUrl: "ws://bad" }), }); - expect(result.status).toBe(400); - const body = (await result.json()) as { error: string }; - expect(body.error).toContain("cdpUrl"); - }); + expect(createBadRemote.status).toBe(400); + const createBadRemoteBody = (await createBadRemote.json()) as { error: string }; + expect(createBadRemoteBody.error).toContain("cdpUrl"); - it("DELETE /profiles/:name returns 404 for non-existent profile", async () => { - const { startBrowserControlServerFromConfig } = await import("./server.js"); - await startBrowserControlServerFromConfig(); - const base = `http://127.0.0.1:${testPort}`; - - const result = await realFetch(`${base}/profiles/nonexistent`, { + const deleteMissing = await realFetch(`${base}/profiles/nonexistent`, { method: "DELETE", }); - expect(result.status).toBe(404); - const body = (await result.json()) as { error: string }; - expect(body.error).toContain("not found"); - }); + expect(deleteMissing.status).toBe(404); + const deleteMissingBody = (await deleteMissing.json()) as { error: string }; + expect(deleteMissingBody.error).toContain("not found"); - it("DELETE /profiles/:name returns 400 for default profile deletion", async () => { - const { startBrowserControlServerFromConfig } = await import("./server.js"); - await startBrowserControlServerFromConfig(); - const base = `http://127.0.0.1:${testPort}`; - - // openclaw is the default profile - const result = await realFetch(`${base}/profiles/openclaw`, { + const deleteDefault = await realFetch(`${base}/profiles/openclaw`, { method: "DELETE", }); - expect(result.status).toBe(400); - const body = (await result.json()) as { error: string }; - expect(body.error).toContain("cannot delete the default profile"); - }); + expect(deleteDefault.status).toBe(400); + const deleteDefaultBody = (await deleteDefault.json()) as { error: string }; + expect(deleteDefaultBody.error).toContain("cannot delete the default profile"); - it("DELETE /profiles/:name returns 400 for invalid name format", async () => { - const { startBrowserControlServerFromConfig } = await import("./server.js"); - await startBrowserControlServerFromConfig(); - const base = `http://127.0.0.1:${testPort}`; - - const result = await realFetch(`${base}/profiles/Invalid-Name!`, { + const deleteInvalid = await realFetch(`${base}/profiles/Invalid-Name!`, { method: "DELETE", }); - expect(result.status).toBe(400); - const body = (await result.json()) as { error: string }; - expect(body.error).toContain("invalid profile name"); + expect(deleteInvalid.status).toBe(400); + const deleteInvalidBody = (await deleteInvalid.json()) as { error: string }; + expect(deleteInvalidBody.error).toContain("invalid profile name"); }); }); diff --git a/src/browser/server.serves-status-starts-browser-requested.test.ts b/src/browser/server.serves-status-starts-browser-requested.test.ts deleted file mode 100644 index df9deed4a5c..00000000000 --- a/src/browser/server.serves-status-starts-browser-requested.test.ts +++ /dev/null @@ -1,329 +0,0 @@ -import { type AddressInfo, createServer } from "node:net"; -import { fetch as realFetch } from "undici"; -import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; - -let testPort = 0; -let _cdpBaseUrl = ""; -let reachable = false; -let cfgAttachOnly = false; -let createTargetId: string | null = null; -let prevGatewayPort: string | undefined; - -const cdpMocks = vi.hoisted(() => ({ - createTargetViaCdp: vi.fn(async () => { - throw new Error("cdp disabled"); - }), - snapshotAria: vi.fn(async () => ({ - nodes: [{ ref: "1", role: "link", name: "x", depth: 0 }], - })), -})); - -const pwMocks = vi.hoisted(() => ({ - armDialogViaPlaywright: vi.fn(async () => {}), - armFileUploadViaPlaywright: vi.fn(async () => {}), - clickViaPlaywright: vi.fn(async () => {}), - closePageViaPlaywright: vi.fn(async () => {}), - closePlaywrightBrowserConnection: vi.fn(async () => {}), - downloadViaPlaywright: vi.fn(async () => ({ - url: "https://example.com/report.pdf", - suggestedFilename: "report.pdf", - path: "/tmp/report.pdf", - })), - dragViaPlaywright: vi.fn(async () => {}), - evaluateViaPlaywright: vi.fn(async () => "ok"), - fillFormViaPlaywright: vi.fn(async () => {}), - getConsoleMessagesViaPlaywright: vi.fn(async () => []), - hoverViaPlaywright: vi.fn(async () => {}), - scrollIntoViewViaPlaywright: vi.fn(async () => {}), - navigateViaPlaywright: vi.fn(async () => ({ url: "https://example.com" })), - pdfViaPlaywright: vi.fn(async () => ({ buffer: Buffer.from("pdf") })), - pressKeyViaPlaywright: vi.fn(async () => {}), - responseBodyViaPlaywright: vi.fn(async () => ({ - url: "https://example.com/api/data", - status: 200, - headers: { "content-type": "application/json" }, - body: '{"ok":true}', - })), - resizeViewportViaPlaywright: vi.fn(async () => {}), - selectOptionViaPlaywright: vi.fn(async () => {}), - setInputFilesViaPlaywright: vi.fn(async () => {}), - snapshotAiViaPlaywright: vi.fn(async () => ({ snapshot: "ok" })), - takeScreenshotViaPlaywright: vi.fn(async () => ({ - buffer: Buffer.from("png"), - })), - typeViaPlaywright: vi.fn(async () => {}), - waitForDownloadViaPlaywright: vi.fn(async () => ({ - url: "https://example.com/report.pdf", - suggestedFilename: "report.pdf", - path: "/tmp/report.pdf", - })), - waitForViaPlaywright: vi.fn(async () => {}), -})); - -function makeProc(pid = 123) { - const handlers = new Map void>>(); - return { - pid, - killed: false, - exitCode: null as number | null, - on: (event: string, cb: (...args: unknown[]) => void) => { - handlers.set(event, [...(handlers.get(event) ?? []), cb]); - return undefined; - }, - emitExit: () => { - for (const cb of handlers.get("exit") ?? []) { - cb(0); - } - }, - kill: () => { - return true; - }, - }; -} - -const proc = makeProc(); - -vi.mock("../config/config.js", async (importOriginal) => { - const actual = await importOriginal(); - return { - ...actual, - loadConfig: () => ({ - browser: { - enabled: true, - color: "#FF4500", - attachOnly: cfgAttachOnly, - headless: true, - defaultProfile: "openclaw", - profiles: { - openclaw: { cdpPort: testPort + 1, color: "#FF4500" }, - }, - }, - }), - writeConfigFile: vi.fn(async () => {}), - }; -}); - -const launchCalls = vi.hoisted(() => [] as Array<{ port: number }>); -vi.mock("./chrome.js", () => ({ - isChromeCdpReady: vi.fn(async () => reachable), - isChromeReachable: vi.fn(async () => reachable), - launchOpenClawChrome: vi.fn(async (_resolved: unknown, profile: { cdpPort: number }) => { - launchCalls.push({ port: profile.cdpPort }); - reachable = true; - return { - pid: 123, - exe: { kind: "chrome", path: "/fake/chrome" }, - userDataDir: "/tmp/openclaw", - cdpPort: profile.cdpPort, - startedAt: Date.now(), - proc, - }; - }), - resolveOpenClawUserDataDir: vi.fn(() => "/tmp/openclaw"), - stopOpenClawChrome: vi.fn(async () => { - reachable = false; - }), -})); - -vi.mock("./cdp.js", () => ({ - createTargetViaCdp: cdpMocks.createTargetViaCdp, - normalizeCdpWsUrl: vi.fn((wsUrl: string) => wsUrl), - snapshotAria: cdpMocks.snapshotAria, - getHeadersWithAuth: vi.fn(() => ({})), - appendCdpPath: vi.fn((cdpUrl: string, path: string) => { - const base = cdpUrl.replace(/\/$/, ""); - const suffix = path.startsWith("/") ? path : `/${path}`; - return `${base}${suffix}`; - }), -})); - -vi.mock("./pw-ai.js", () => pwMocks); - -vi.mock("../media/store.js", () => ({ - ensureMediaDir: vi.fn(async () => {}), - saveMediaBuffer: vi.fn(async () => ({ path: "/tmp/fake.png" })), -})); - -vi.mock("./screenshot.js", () => ({ - DEFAULT_BROWSER_SCREENSHOT_MAX_BYTES: 128, - DEFAULT_BROWSER_SCREENSHOT_MAX_SIDE: 64, - normalizeBrowserScreenshot: vi.fn(async (buf: Buffer) => ({ - buffer: buf, - contentType: "image/png", - })), -})); - -async function getFreePort(): Promise { - while (true) { - const port = await new Promise((resolve, reject) => { - const s = createServer(); - s.once("error", reject); - s.listen(0, "127.0.0.1", () => { - const assigned = (s.address() as AddressInfo).port; - s.close((err) => (err ? reject(err) : resolve(assigned))); - }); - }); - if (port < 65535) { - return port; - } - } -} - -function makeResponse( - body: unknown, - init?: { ok?: boolean; status?: number; text?: string }, -): Response { - const ok = init?.ok ?? true; - const status = init?.status ?? 200; - const text = init?.text ?? ""; - return { - ok, - status, - json: async () => body, - text: async () => text, - } as unknown as Response; -} - -describe("browser control server", () => { - beforeEach(async () => { - reachable = false; - cfgAttachOnly = false; - createTargetId = null; - - cdpMocks.createTargetViaCdp.mockImplementation(async () => { - if (createTargetId) { - return { targetId: createTargetId }; - } - throw new Error("cdp disabled"); - }); - - for (const fn of Object.values(pwMocks)) { - fn.mockClear(); - } - for (const fn of Object.values(cdpMocks)) { - fn.mockClear(); - } - - testPort = await getFreePort(); - _cdpBaseUrl = `http://127.0.0.1:${testPort + 1}`; - prevGatewayPort = process.env.OPENCLAW_GATEWAY_PORT; - process.env.OPENCLAW_GATEWAY_PORT = String(testPort - 2); - - // Minimal CDP JSON endpoints used by the server. - let putNewCalls = 0; - vi.stubGlobal( - "fetch", - vi.fn(async (url: string, init?: RequestInit) => { - const u = String(url); - if (u.includes("/json/list")) { - if (!reachable) { - return makeResponse([]); - } - return makeResponse([ - { - id: "abcd1234", - title: "Tab", - url: "https://example.com", - webSocketDebuggerUrl: "ws://127.0.0.1/devtools/page/abcd1234", - type: "page", - }, - { - id: "abce9999", - title: "Other", - url: "https://other", - webSocketDebuggerUrl: "ws://127.0.0.1/devtools/page/abce9999", - type: "page", - }, - ]); - } - if (u.includes("/json/new?")) { - if (init?.method === "PUT") { - putNewCalls += 1; - if (putNewCalls === 1) { - return makeResponse({}, { ok: false, status: 405, text: "" }); - } - } - return makeResponse({ - id: "newtab1", - title: "", - url: "about:blank", - webSocketDebuggerUrl: "ws://127.0.0.1/devtools/page/newtab1", - type: "page", - }); - } - if (u.includes("/json/activate/")) { - return makeResponse("ok"); - } - if (u.includes("/json/close/")) { - return makeResponse("ok"); - } - return makeResponse({}, { ok: false, status: 500, text: "unexpected" }); - }), - ); - }); - - afterEach(async () => { - vi.unstubAllGlobals(); - vi.restoreAllMocks(); - if (prevGatewayPort === undefined) { - delete process.env.OPENCLAW_GATEWAY_PORT; - } else { - process.env.OPENCLAW_GATEWAY_PORT = prevGatewayPort; - } - const { stopBrowserControlServer } = await import("./server.js"); - await stopBrowserControlServer(); - }); - - it("serves status + starts browser when requested", async () => { - const { startBrowserControlServerFromConfig } = await import("./server.js"); - const started = await startBrowserControlServerFromConfig(); - expect(started?.port).toBe(testPort); - - const base = `http://127.0.0.1:${testPort}`; - const s1 = (await realFetch(`${base}/`).then((r) => r.json())) as { - running: boolean; - pid: number | null; - }; - expect(s1.running).toBe(false); - expect(s1.pid).toBe(null); - - await realFetch(`${base}/start`, { method: "POST" }).then((r) => r.json()); - const s2 = (await realFetch(`${base}/`).then((r) => r.json())) as { - running: boolean; - pid: number | null; - chosenBrowser: string | null; - }; - expect(s2.running).toBe(true); - expect(s2.pid).toBe(123); - expect(s2.chosenBrowser).toBe("chrome"); - expect(launchCalls.length).toBeGreaterThan(0); - }); - - it("handles tabs: list, open, focus conflict on ambiguous prefix", async () => { - const { startBrowserControlServerFromConfig } = await import("./server.js"); - await startBrowserControlServerFromConfig(); - const base = `http://127.0.0.1:${testPort}`; - - await realFetch(`${base}/start`, { method: "POST" }).then((r) => r.json()); - const tabs = (await realFetch(`${base}/tabs`).then((r) => r.json())) as { - running: boolean; - tabs: Array<{ targetId: string }>; - }; - expect(tabs.running).toBe(true); - expect(tabs.tabs.length).toBeGreaterThan(0); - - const opened = await realFetch(`${base}/tabs/open`, { - method: "POST", - headers: { "Content-Type": "application/json" }, - body: JSON.stringify({ url: "https://example.com" }), - }).then((r) => r.json()); - expect(opened).toMatchObject({ targetId: "newtab1" }); - - const focus = await realFetch(`${base}/tabs/focus`, { - method: "POST", - headers: { "Content-Type": "application/json" }, - body: JSON.stringify({ targetId: "abc" }), - }); - expect(focus.status).toBe(409); - }); -}); diff --git a/src/browser/server.skips-default-maxchars-explicitly-set-zero.test.ts b/src/browser/server.skips-default-maxchars-explicitly-set-zero.test.ts index 7caa3b292cd..8b5ca4e5802 100644 --- a/src/browser/server.skips-default-maxchars-explicitly-set-zero.test.ts +++ b/src/browser/server.skips-default-maxchars-explicitly-set-zero.test.ts @@ -1,12 +1,11 @@ import { type AddressInfo, createServer } from "node:net"; import { fetch as realFetch } from "undici"; -import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; let testPort = 0; let cdpBaseUrl = ""; let reachable = false; let cfgAttachOnly = false; -let createTargetId: string | null = null; let prevGatewayPort: string | undefined; const cdpMocks = vi.hoisted(() => ({ @@ -185,15 +184,12 @@ function makeResponse( } describe("browser control server", () => { - beforeEach(async () => { + beforeAll(async () => { reachable = false; cfgAttachOnly = false; - createTargetId = null; + launchCalls.length = 0; cdpMocks.createTargetViaCdp.mockImplementation(async () => { - if (createTargetId) { - return { targetId: createTargetId }; - } throw new Error("cdp disabled"); }); @@ -210,7 +206,6 @@ describe("browser control server", () => { process.env.OPENCLAW_GATEWAY_PORT = String(testPort - 2); // Minimal CDP JSON endpoints used by the server. - let putNewCalls = 0; vi.stubGlobal( "fetch", vi.fn(async (url: string, init?: RequestInit) => { @@ -236,33 +231,13 @@ describe("browser control server", () => { }, ]); } - if (u.includes("/json/new?")) { - if (init?.method === "PUT") { - putNewCalls += 1; - if (putNewCalls === 1) { - return makeResponse({}, { ok: false, status: 405, text: "" }); - } - } - return makeResponse({ - id: "newtab1", - title: "", - url: "about:blank", - webSocketDebuggerUrl: "ws://127.0.0.1/devtools/page/newtab1", - type: "page", - }); - } - if (u.includes("/json/activate/")) { - return makeResponse("ok"); - } - if (u.includes("/json/close/")) { - return makeResponse("ok"); - } + void init; return makeResponse({}, { ok: false, status: 500, text: "unexpected" }); }), ); }); - afterEach(async () => { + afterAll(async () => { vi.unstubAllGlobals(); vi.restoreAllMocks(); if (prevGatewayPort === undefined) { @@ -274,190 +249,52 @@ describe("browser control server", () => { await stopBrowserControlServer(); }); - it("skips default maxChars when explicitly set to zero", async () => { + it("covers primary control routes, validation, and profile compatibility", async () => { const { startBrowserControlServerFromConfig } = await import("./server.js"); - await startBrowserControlServerFromConfig(); + const started = await startBrowserControlServerFromConfig(); + expect(started?.port).toBe(testPort); + const base = `http://127.0.0.1:${testPort}`; - await realFetch(`${base}/start`, { method: "POST" }).then((r) => r.json()); + const statusBeforeStart = (await realFetch(`${base}/`).then((r) => r.json())) as { + running: boolean; + pid: number | null; + }; + expect(statusBeforeStart.running).toBe(false); + expect(statusBeforeStart.pid).toBe(null); + expect(statusBeforeStart.profile).toBe("openclaw"); + + const startedPayload = (await realFetch(`${base}/start`, { method: "POST" }).then((r) => + r.json(), + )) as { ok: boolean; profile?: string }; + expect(startedPayload.ok).toBe(true); + expect(startedPayload.profile).toBe("openclaw"); + + const statusAfterStart = (await realFetch(`${base}/`).then((r) => r.json())) as { + running: boolean; + pid: number | null; + chosenBrowser: string | null; + }; + expect(statusAfterStart.running).toBe(true); + expect(statusAfterStart.pid).toBe(123); + expect(statusAfterStart.chosenBrowser).toBe("chrome"); + expect(launchCalls.length).toBeGreaterThan(0); const snapAi = (await realFetch(`${base}/snapshot?format=ai&maxChars=0`).then((r) => r.json(), )) as { ok: boolean; format?: string }; expect(snapAi.ok).toBe(true); expect(snapAi.format).toBe("ai"); - const [call] = pwMocks.snapshotAiViaPlaywright.mock.calls.at(-1) ?? []; expect(call).toEqual({ cdpUrl: cdpBaseUrl, targetId: "abcd1234", }); - }); - it("validates agent inputs (agent routes)", async () => { - const { startBrowserControlServerFromConfig } = await import("./server.js"); - await startBrowserControlServerFromConfig(); - const base = `http://127.0.0.1:${testPort}`; - await realFetch(`${base}/start`, { method: "POST" }).then((r) => r.json()); - - const navMissing = await realFetch(`${base}/navigate`, { - method: "POST", - headers: { "Content-Type": "application/json" }, - body: JSON.stringify({}), - }); - expect(navMissing.status).toBe(400); - - const actMissing = await realFetch(`${base}/act`, { - method: "POST", - headers: { "Content-Type": "application/json" }, - body: JSON.stringify({}), - }); - expect(actMissing.status).toBe(400); - - const clickMissingRef = await realFetch(`${base}/act`, { - method: "POST", - headers: { "Content-Type": "application/json" }, - body: JSON.stringify({ kind: "click" }), - }); - expect(clickMissingRef.status).toBe(400); - - const scrollMissingRef = await realFetch(`${base}/act`, { - method: "POST", - headers: { "Content-Type": "application/json" }, - body: JSON.stringify({ kind: "scrollIntoView" }), - }); - expect(scrollMissingRef.status).toBe(400); - - const scrollSelectorUnsupported = await realFetch(`${base}/act`, { - method: "POST", - headers: { "Content-Type": "application/json" }, - body: JSON.stringify({ kind: "scrollIntoView", selector: "button.save" }), - }); - expect(scrollSelectorUnsupported.status).toBe(400); - - const clickBadButton = await realFetch(`${base}/act`, { - method: "POST", - headers: { "Content-Type": "application/json" }, - body: JSON.stringify({ kind: "click", ref: "1", button: "nope" }), - }); - expect(clickBadButton.status).toBe(400); - - const clickBadModifiers = await realFetch(`${base}/act`, { - method: "POST", - headers: { "Content-Type": "application/json" }, - body: JSON.stringify({ kind: "click", ref: "1", modifiers: ["Nope"] }), - }); - expect(clickBadModifiers.status).toBe(400); - - const typeBadText = await realFetch(`${base}/act`, { - method: "POST", - headers: { "Content-Type": "application/json" }, - body: JSON.stringify({ kind: "type", ref: "1", text: 123 }), - }); - expect(typeBadText.status).toBe(400); - - const uploadMissingPaths = await realFetch(`${base}/hooks/file-chooser`, { - method: "POST", - headers: { "Content-Type": "application/json" }, - body: JSON.stringify({}), - }); - expect(uploadMissingPaths.status).toBe(400); - - const dialogMissingAccept = await realFetch(`${base}/hooks/dialog`, { - method: "POST", - headers: { "Content-Type": "application/json" }, - body: JSON.stringify({}), - }); - expect(dialogMissingAccept.status).toBe(400); - - const snapDefault = (await realFetch(`${base}/snapshot?format=wat`).then((r) => r.json())) as { + const stopped = (await realFetch(`${base}/stop`, { method: "POST" }).then((r) => r.json())) as { ok: boolean; - format?: string; + profile?: string; }; - expect(snapDefault.ok).toBe(true); - expect(snapDefault.format).toBe("ai"); - - const screenshotBadCombo = await realFetch(`${base}/screenshot`, { - method: "POST", - headers: { "Content-Type": "application/json" }, - body: JSON.stringify({ fullPage: true, element: "body" }), - }); - expect(screenshotBadCombo.status).toBe(400); - }); - - it("covers common error branches", async () => { - cfgAttachOnly = true; - const { startBrowserControlServerFromConfig } = await import("./server.js"); - await startBrowserControlServerFromConfig(); - const base = `http://127.0.0.1:${testPort}`; - - const missing = await realFetch(`${base}/tabs/open`, { - method: "POST", - headers: { "Content-Type": "application/json" }, - body: JSON.stringify({}), - }); - expect(missing.status).toBe(400); - - reachable = false; - const started = (await realFetch(`${base}/start`, { - method: "POST", - }).then((r) => r.json())) as { error?: string }; - expect(started.error ?? "").toMatch(/attachOnly/i); - }); - - it("allows attachOnly servers to ensure reachability via callback", async () => { - cfgAttachOnly = true; - reachable = false; - const { startBrowserBridgeServer } = await import("./bridge-server.js"); - - const ensured = vi.fn(async () => { - reachable = true; - }); - - const bridge = await startBrowserBridgeServer({ - resolved: { - enabled: true, - controlPort: 0, - cdpProtocol: "http", - cdpHost: "127.0.0.1", - cdpIsLoopback: true, - color: "#FF4500", - headless: true, - noSandbox: false, - attachOnly: true, - defaultProfile: "openclaw", - profiles: { - openclaw: { cdpPort: testPort + 1, color: "#FF4500" }, - }, - }, - onEnsureAttachTarget: ensured, - }); - - const started = (await realFetch(`${bridge.baseUrl}/start`, { - method: "POST", - }).then((r) => r.json())) as { ok?: boolean; error?: string }; - expect(started.error).toBeUndefined(); - expect(started.ok).toBe(true); - const status = (await realFetch(`${bridge.baseUrl}/`).then((r) => r.json())) as { - running?: boolean; - }; - expect(status.running).toBe(true); - expect(ensured).toHaveBeenCalledTimes(1); - - await new Promise((resolve) => bridge.server.close(() => resolve())); - }); - - it("opens tabs via CDP createTarget path", async () => { - const { startBrowserControlServerFromConfig } = await import("./server.js"); - await startBrowserControlServerFromConfig(); - const base = `http://127.0.0.1:${testPort}`; - await realFetch(`${base}/start`, { method: "POST" }).then((r) => r.json()); - - createTargetId = "abcd1234"; - const opened = (await realFetch(`${base}/tabs/open`, { - method: "POST", - headers: { "Content-Type": "application/json" }, - body: JSON.stringify({ url: "https://example.com" }), - }).then((r) => r.json())) as { targetId?: string }; - expect(opened.targetId).toBe("abcd1234"); + expect(stopped.ok).toBe(true); + expect(stopped.profile).toBe("openclaw"); }); }); diff --git a/src/browser/server.ts b/src/browser/server.ts index 2f734f031d5..03f084f168d 100644 --- a/src/browser/server.ts +++ b/src/browser/server.ts @@ -7,8 +7,13 @@ import { safeEqualSecret } from "../security/secret-equal.js"; import { resolveBrowserConfig, resolveProfile } from "./config.js"; import { ensureBrowserControlAuth, resolveBrowserControlAuth } from "./control-auth.js"; import { ensureChromeExtensionRelayServer } from "./extension-relay.js"; +import { isPwAiLoaded } from "./pw-ai-state.js"; import { registerBrowserRoutes } from "./routes/index.js"; -import { type BrowserServerState, createBrowserRouteContext } from "./server-context.js"; +import { + type BrowserServerState, + createBrowserRouteContext, + listKnownProfileNames, +} from "./server-context.js"; let state: BrowserServerState | null = null; const log = createSubsystemLogger("browser"); @@ -124,6 +129,7 @@ export async function startBrowserControlServerFromConfig(): Promise state, + refreshConfigFromDisk: true, }); registerBrowserRoutes(app as unknown as BrowserRouteRegistrar, ctx); @@ -172,12 +178,13 @@ export async function stopBrowserControlServer(): Promise { const ctx = createBrowserRouteContext({ getState: () => state, + refreshConfigFromDisk: true, }); try { const current = state; if (current) { - for (const name of Object.keys(current.resolved.profiles)) { + for (const name of listKnownProfileNames(current)) { try { await ctx.forProfile(name).stopRunningBrowser(); } catch { @@ -196,11 +203,13 @@ export async function stopBrowserControlServer(): Promise { } state = null; - // Optional: Playwright is not always available (e.g. embedded gateway builds). - try { - const mod = await import("./pw-ai.js"); - await mod.closePlaywrightBrowserConnection(); - } catch { - // ignore + // Optional: avoid importing heavy Playwright bridge when this process never used it. + if (isPwAiLoaded()) { + try { + const mod = await import("./pw-ai.js"); + await mod.closePlaywrightBrowserConnection(); + } catch { + // ignore + } } } diff --git a/src/canvas-host/server.test.ts b/src/canvas-host/server.test.ts index b768aa02b4d..c5adec8f94c 100644 --- a/src/canvas-host/server.test.ts +++ b/src/canvas-host/server.test.ts @@ -3,7 +3,7 @@ import fs from "node:fs/promises"; import { createServer } from "node:http"; import os from "node:os"; import path from "node:path"; -import { describe, expect, it, vi } from "vitest"; +import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; import { WebSocket } from "ws"; import { rawDataToString } from "../infra/ws.js"; import { defaultRuntime } from "../runtime.js"; @@ -11,6 +11,27 @@ import { A2UI_PATH, CANVAS_HOST_PATH, CANVAS_WS_PATH, injectCanvasLiveReload } f import { createCanvasHostHandler, startCanvasHost } from "./server.js"; describe("canvas host", () => { + const quietRuntime = { + ...defaultRuntime, + log: (..._args: Parameters) => {}, + }; + let fixtureRoot = ""; + let fixtureCount = 0; + + const createCaseDir = async () => { + const dir = path.join(fixtureRoot, `case-${fixtureCount++}`); + await fs.mkdir(dir, { recursive: true }); + return dir; + }; + + beforeAll(async () => { + fixtureRoot = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-canvas-fixtures-")); + }); + + afterAll(async () => { + await fs.rm(fixtureRoot, { recursive: true, force: true }); + }); + it("injects live reload script", () => { const out = injectCanvasLiveReload("Hello"); expect(out).toContain(CANVAS_WS_PATH); @@ -20,10 +41,10 @@ describe("canvas host", () => { }); it("creates a default index.html when missing", async () => { - const dir = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-canvas-")); + const dir = await createCaseDir(); const server = await startCanvasHost({ - runtime: defaultRuntime, + runtime: quietRuntime, rootDir: dir, port: 0, listenHost: "127.0.0.1", @@ -39,16 +60,15 @@ describe("canvas host", () => { expect(html).toContain(CANVAS_WS_PATH); } finally { await server.close(); - await fs.rm(dir, { recursive: true, force: true }); } }); it("skips live reload injection when disabled", async () => { - const dir = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-canvas-")); + const dir = await createCaseDir(); await fs.writeFile(path.join(dir, "index.html"), "no-reload", "utf8"); const server = await startCanvasHost({ - runtime: defaultRuntime, + runtime: quietRuntime, rootDir: dir, port: 0, listenHost: "127.0.0.1", @@ -67,16 +87,15 @@ describe("canvas host", () => { expect(wsRes.status).toBe(404); } finally { await server.close(); - await fs.rm(dir, { recursive: true, force: true }); } }); - it("serves canvas content from the mounted base path", async () => { - const dir = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-canvas-")); + it("serves canvas content from the mounted base path and reuses handlers without double close", async () => { + const dir = await createCaseDir(); await fs.writeFile(path.join(dir, "index.html"), "v1", "utf8"); const handler = await createCanvasHostHandler({ - runtime: defaultRuntime, + runtime: quietRuntime, rootDir: dir, basePath: CANVAS_HOST_PATH, allowInTests: true, @@ -112,30 +131,16 @@ describe("canvas host", () => { const miss = await fetch(`http://127.0.0.1:${port}/`); expect(miss.status).toBe(404); } finally { - await handler.close(); await new Promise((resolve, reject) => server.close((err) => (err ? reject(err) : resolve())), ); - await fs.rm(dir, { recursive: true, force: true }); } - }); - - it("reuses a handler without closing it twice", async () => { - const dir = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-canvas-")); - await fs.writeFile(path.join(dir, "index.html"), "v1", "utf8"); - - const handler = await createCanvasHostHandler({ - runtime: defaultRuntime, - rootDir: dir, - basePath: CANVAS_HOST_PATH, - allowInTests: true, - }); const originalClose = handler.close; const closeSpy = vi.fn(async () => originalClose()); handler.close = closeSpy; - const server = await startCanvasHost({ - runtime: defaultRuntime, + const hosted = await startCanvasHost({ + runtime: quietRuntime, handler, ownsHandler: false, port: 0, @@ -144,22 +149,21 @@ describe("canvas host", () => { }); try { - expect(server.port).toBeGreaterThan(0); + expect(hosted.port).toBeGreaterThan(0); } finally { - await server.close(); + await hosted.close(); expect(closeSpy).not.toHaveBeenCalled(); await originalClose(); - await fs.rm(dir, { recursive: true, force: true }); } }); it("serves HTML with injection and broadcasts reload on file changes", async () => { - const dir = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-canvas-")); + const dir = await createCaseDir(); const index = path.join(dir, "index.html"); await fs.writeFile(index, "v1", "utf8"); const server = await startCanvasHost({ - runtime: defaultRuntime, + runtime: quietRuntime, rootDir: dir, port: 0, listenHost: "127.0.0.1", @@ -194,21 +198,22 @@ describe("canvas host", () => { }); }); - await new Promise((resolve) => setTimeout(resolve, 100)); await fs.writeFile(index, "v2", "utf8"); expect(await msg).toBe("reload"); ws.close(); } finally { await server.close(); - await fs.rm(dir, { recursive: true, force: true }); } }, 20_000); - it("serves the gateway-hosted A2UI scaffold", async () => { - const dir = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-canvas-")); + it("serves A2UI scaffold and blocks traversal/symlink escapes", async () => { + const dir = await createCaseDir(); const a2uiRoot = path.resolve(process.cwd(), "src/canvas-host/a2ui"); const bundlePath = path.join(a2uiRoot, "a2ui.bundle.js"); + const linkName = `test-link-${Date.now()}-${Math.random().toString(16).slice(2)}.txt`; + const linkPath = path.join(a2uiRoot, linkName); let createdBundle = false; + let createdLink = false; try { await fs.stat(bundlePath); @@ -217,8 +222,11 @@ describe("canvas host", () => { createdBundle = true; } + await fs.symlink(path.join(process.cwd(), "package.json"), linkPath); + createdLink = true; + const server = await startCanvasHost({ - runtime: defaultRuntime, + runtime: quietRuntime, rootDir: dir, port: 0, listenHost: "127.0.0.1", @@ -238,80 +246,14 @@ describe("canvas host", () => { const js = await bundleRes.text(); expect(bundleRes.status).toBe(200); expect(js).toContain("openclawA2UI"); - } finally { - await server.close(); - if (createdBundle) { - await fs.rm(bundlePath, { force: true }); - } - await fs.rm(dir, { recursive: true, force: true }); - } - }); - - it("rejects traversal-style A2UI asset requests", async () => { - const dir = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-canvas-")); - const a2uiRoot = path.resolve(process.cwd(), "src/canvas-host/a2ui"); - const bundlePath = path.join(a2uiRoot, "a2ui.bundle.js"); - let createdBundle = false; - - try { - await fs.stat(bundlePath); - } catch { - await fs.writeFile(bundlePath, "window.openclawA2UI = {};", "utf8"); - createdBundle = true; - } - - const server = await startCanvasHost({ - runtime: defaultRuntime, - rootDir: dir, - port: 0, - listenHost: "127.0.0.1", - allowInTests: true, - }); - - try { - const res = await fetch(`http://127.0.0.1:${server.port}${A2UI_PATH}/%2e%2e%2fpackage.json`); - expect(res.status).toBe(404); - expect(await res.text()).toBe("not found"); - } finally { - await server.close(); - if (createdBundle) { - await fs.rm(bundlePath, { force: true }); - } - await fs.rm(dir, { recursive: true, force: true }); - } - }); - - it("rejects A2UI symlink escapes", async () => { - const dir = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-canvas-")); - const a2uiRoot = path.resolve(process.cwd(), "src/canvas-host/a2ui"); - const bundlePath = path.join(a2uiRoot, "a2ui.bundle.js"); - const linkName = `test-link-${Date.now()}-${Math.random().toString(16).slice(2)}.txt`; - const linkPath = path.join(a2uiRoot, linkName); - let createdBundle = false; - let createdLink = false; - - try { - await fs.stat(bundlePath); - } catch { - await fs.writeFile(bundlePath, "window.openclawA2UI = {};", "utf8"); - createdBundle = true; - } - - await fs.symlink(path.join(process.cwd(), "package.json"), linkPath); - createdLink = true; - - const server = await startCanvasHost({ - runtime: defaultRuntime, - rootDir: dir, - port: 0, - listenHost: "127.0.0.1", - allowInTests: true, - }); - - try { - const res = await fetch(`http://127.0.0.1:${server.port}${A2UI_PATH}/${linkName}`); - expect(res.status).toBe(404); - expect(await res.text()).toBe("not found"); + const traversalRes = await fetch( + `http://127.0.0.1:${server.port}${A2UI_PATH}/%2e%2e%2fpackage.json`, + ); + expect(traversalRes.status).toBe(404); + expect(await traversalRes.text()).toBe("not found"); + const symlinkRes = await fetch(`http://127.0.0.1:${server.port}${A2UI_PATH}/${linkName}`); + expect(symlinkRes.status).toBe(404); + expect(await symlinkRes.text()).toBe("not found"); } finally { await server.close(); if (createdLink) { @@ -320,7 +262,6 @@ describe("canvas host", () => { if (createdBundle) { await fs.rm(bundlePath, { force: true }); } - await fs.rm(dir, { recursive: true, force: true }); } }); }); diff --git a/src/canvas-host/server.ts b/src/canvas-host/server.ts index 65b0b83a4cd..3dfc83af410 100644 --- a/src/canvas-host/server.ts +++ b/src/canvas-host/server.ts @@ -264,6 +264,10 @@ export async function createCanvasHostHandler( const rootReal = await prepareCanvasRoot(rootDir); const liveReload = opts.liveReload !== false; + const testMode = opts.allowInTests === true; + const reloadDebounceMs = testMode ? 12 : 75; + const writeStabilityThresholdMs = testMode ? 12 : 75; + const writePollIntervalMs = testMode ? 5 : 10; const wss = liveReload ? new WebSocketServer({ noServer: true }) : null; const sockets = new Set(); if (wss) { @@ -293,7 +297,7 @@ export async function createCanvasHostHandler( debounce = setTimeout(() => { debounce = null; broadcastReload(); - }, 75); + }, reloadDebounceMs); debounce.unref?.(); }; @@ -301,8 +305,11 @@ export async function createCanvasHostHandler( const watcher = liveReload ? chokidar.watch(rootReal, { ignoreInitial: true, - awaitWriteFinish: { stabilityThreshold: 75, pollInterval: 10 }, - usePolling: opts.allowInTests === true, + awaitWriteFinish: { + stabilityThreshold: writeStabilityThresholdMs, + pollInterval: writePollIntervalMs, + }, + usePolling: testMode, ignored: [ /(^|[\\/])\../, // dotfiles /(^|[\\/])node_modules([\\/]|$)/, diff --git a/src/channels/plugins/actions/discord.test.ts b/src/channels/plugins/actions/discord.test.ts index 7c41cda9d61..fc30a0a7566 100644 --- a/src/channels/plugins/actions/discord.test.ts +++ b/src/channels/plugins/actions/discord.test.ts @@ -21,20 +21,12 @@ vi.mock("../../../discord/send.js", async () => { }; }); -const loadHandleDiscordMessageAction = async () => { - const mod = await import("./discord/handle-action.js"); - return mod.handleDiscordMessageAction; -}; - -const loadDiscordMessageActions = async () => { - const mod = await import("./discord.js"); - return mod.discordMessageActions; -}; +const { handleDiscordMessageAction } = await import("./discord/handle-action.js"); +const { discordMessageActions } = await import("./discord.js"); describe("discord message actions", () => { it("lists channel and upload actions by default", async () => { const cfg = { channels: { discord: { token: "d0" } } } as OpenClawConfig; - const discordMessageActions = await loadDiscordMessageActions(); const actions = discordMessageActions.listActions?.({ cfg }) ?? []; expect(actions).toContain("emoji-upload"); @@ -46,7 +38,6 @@ describe("discord message actions", () => { const cfg = { channels: { discord: { token: "d0", actions: { channels: false } } }, } as OpenClawConfig; - const discordMessageActions = await loadDiscordMessageActions(); const actions = discordMessageActions.listActions?.({ cfg }) ?? []; expect(actions).not.toContain("channel-create"); @@ -56,7 +47,6 @@ describe("discord message actions", () => { describe("handleDiscordMessageAction", () => { it("forwards context accountId for send", async () => { sendMessageDiscord.mockClear(); - const handleDiscordMessageAction = await loadHandleDiscordMessageAction(); await handleDiscordMessageAction({ action: "send", @@ -79,7 +69,6 @@ describe("handleDiscordMessageAction", () => { it("falls back to params accountId when context missing", async () => { sendPollDiscord.mockClear(); - const handleDiscordMessageAction = await loadHandleDiscordMessageAction(); await handleDiscordMessageAction({ action: "poll", @@ -106,7 +95,6 @@ describe("handleDiscordMessageAction", () => { it("forwards accountId for thread replies", async () => { sendMessageDiscord.mockClear(); - const handleDiscordMessageAction = await loadHandleDiscordMessageAction(); await handleDiscordMessageAction({ action: "thread-reply", @@ -129,7 +117,6 @@ describe("handleDiscordMessageAction", () => { it("accepts threadId for thread replies (tool compatibility)", async () => { sendMessageDiscord.mockClear(); - const handleDiscordMessageAction = await loadHandleDiscordMessageAction(); await handleDiscordMessageAction({ action: "thread-reply", diff --git a/src/channels/plugins/outbound/slack.test.ts b/src/channels/plugins/outbound/slack.test.ts new file mode 100644 index 00000000000..08863d24b7f --- /dev/null +++ b/src/channels/plugins/outbound/slack.test.ts @@ -0,0 +1,124 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; + +vi.mock("../../../slack/send.js", () => ({ + sendMessageSlack: vi.fn().mockResolvedValue({ ts: "1234.5678", channel: "C123" }), +})); + +vi.mock("../../../plugins/hook-runner-global.js", () => ({ + getGlobalHookRunner: vi.fn(), +})); + +import { getGlobalHookRunner } from "../../../plugins/hook-runner-global.js"; +import { sendMessageSlack } from "../../../slack/send.js"; +import { slackOutbound } from "./slack.js"; + +describe("slack outbound hook wiring", () => { + beforeEach(() => { + vi.clearAllMocks(); + }); + + afterEach(() => { + vi.restoreAllMocks(); + }); + + it("calls send without hooks when no hooks registered", async () => { + vi.mocked(getGlobalHookRunner).mockReturnValue(null); + + await slackOutbound.sendText({ + to: "C123", + text: "hello", + accountId: "default", + replyToId: "1111.2222", + }); + + expect(sendMessageSlack).toHaveBeenCalledWith("C123", "hello", { + threadTs: "1111.2222", + accountId: "default", + }); + }); + + it("calls message_sending hook before sending", async () => { + const mockRunner = { + hasHooks: vi.fn().mockReturnValue(true), + runMessageSending: vi.fn().mockResolvedValue(undefined), + }; + // oxlint-disable-next-line typescript/no-explicit-any + vi.mocked(getGlobalHookRunner).mockReturnValue(mockRunner as any); + + await slackOutbound.sendText({ + to: "C123", + text: "hello", + accountId: "default", + replyToId: "1111.2222", + }); + + expect(mockRunner.hasHooks).toHaveBeenCalledWith("message_sending"); + expect(mockRunner.runMessageSending).toHaveBeenCalledWith( + { to: "C123", content: "hello", metadata: { threadTs: "1111.2222", channelId: "C123" } }, + { channelId: "slack", accountId: "default" }, + ); + expect(sendMessageSlack).toHaveBeenCalledWith("C123", "hello", { + threadTs: "1111.2222", + accountId: "default", + }); + }); + + it("cancels send when hook returns cancel:true", async () => { + const mockRunner = { + hasHooks: vi.fn().mockReturnValue(true), + runMessageSending: vi.fn().mockResolvedValue({ cancel: true }), + }; + // oxlint-disable-next-line typescript/no-explicit-any + vi.mocked(getGlobalHookRunner).mockReturnValue(mockRunner as any); + + const result = await slackOutbound.sendText({ + to: "C123", + text: "hello", + accountId: "default", + replyToId: "1111.2222", + }); + + expect(sendMessageSlack).not.toHaveBeenCalled(); + expect(result.channel).toBe("slack"); + }); + + it("modifies text when hook returns content", async () => { + const mockRunner = { + hasHooks: vi.fn().mockReturnValue(true), + runMessageSending: vi.fn().mockResolvedValue({ content: "modified" }), + }; + // oxlint-disable-next-line typescript/no-explicit-any + vi.mocked(getGlobalHookRunner).mockReturnValue(mockRunner as any); + + await slackOutbound.sendText({ + to: "C123", + text: "original", + accountId: "default", + replyToId: "1111.2222", + }); + + expect(sendMessageSlack).toHaveBeenCalledWith("C123", "modified", { + threadTs: "1111.2222", + accountId: "default", + }); + }); + + it("skips hooks when runner has no message_sending hooks", async () => { + const mockRunner = { + hasHooks: vi.fn().mockReturnValue(false), + runMessageSending: vi.fn(), + }; + // oxlint-disable-next-line typescript/no-explicit-any + vi.mocked(getGlobalHookRunner).mockReturnValue(mockRunner as any); + + await slackOutbound.sendText({ + to: "C123", + text: "hello", + accountId: "default", + replyToId: "1111.2222", + }); + + expect(mockRunner.runMessageSending).not.toHaveBeenCalled(); + expect(sendMessageSlack).toHaveBeenCalled(); + }); +}); diff --git a/src/channels/plugins/outbound/slack.ts b/src/channels/plugins/outbound/slack.ts index 08d27bd7073..dde96245538 100644 --- a/src/channels/plugins/outbound/slack.ts +++ b/src/channels/plugins/outbound/slack.ts @@ -1,4 +1,5 @@ import type { ChannelOutboundAdapter } from "../types.js"; +import { getGlobalHookRunner } from "../../../plugins/hook-runner-global.js"; import { sendMessageSlack } from "../../../slack/send.js"; export const slackOutbound: ChannelOutboundAdapter = { @@ -9,7 +10,29 @@ export const slackOutbound: ChannelOutboundAdapter = { const send = deps?.sendSlack ?? sendMessageSlack; // Use threadId fallback so routed tool notifications stay in the Slack thread. const threadTs = replyToId ?? (threadId != null ? String(threadId) : undefined); - const result = await send(to, text, { + let finalText = text; + + // Run message_sending hooks (e.g. thread-ownership can cancel the send). + const hookRunner = getGlobalHookRunner(); + if (hookRunner?.hasHooks("message_sending")) { + const hookResult = await hookRunner.runMessageSending( + { to, content: text, metadata: { threadTs, channelId: to } }, + { channelId: "slack", accountId: accountId ?? undefined }, + ); + if (hookResult?.cancel) { + return { + channel: "slack", + messageId: "cancelled-by-hook", + channelId: to, + meta: { cancelled: true }, + }; + } + if (hookResult?.content) { + finalText = hookResult.content; + } + } + + const result = await send(to, finalText, { threadTs, accountId: accountId ?? undefined, }); @@ -19,7 +42,29 @@ export const slackOutbound: ChannelOutboundAdapter = { const send = deps?.sendSlack ?? sendMessageSlack; // Use threadId fallback so routed tool notifications stay in the Slack thread. const threadTs = replyToId ?? (threadId != null ? String(threadId) : undefined); - const result = await send(to, text, { + let finalText = text; + + // Run message_sending hooks (e.g. thread-ownership can cancel the send). + const hookRunner = getGlobalHookRunner(); + if (hookRunner?.hasHooks("message_sending")) { + const hookResult = await hookRunner.runMessageSending( + { to, content: text, metadata: { threadTs, channelId: to, mediaUrl } }, + { channelId: "slack", accountId: accountId ?? undefined }, + ); + if (hookResult?.cancel) { + return { + channel: "slack", + messageId: "cancelled-by-hook", + channelId: to, + meta: { cancelled: true }, + }; + } + if (hookResult?.content) { + finalText = hookResult.content; + } + } + + const result = await send(to, finalText, { mediaUrl, threadTs, accountId: accountId ?? undefined, diff --git a/src/cli/acp-cli.ts b/src/cli/acp-cli.ts index 1be77e71fcd..c86deb48f28 100644 --- a/src/cli/acp-cli.ts +++ b/src/cli/acp-cli.ts @@ -22,9 +22,9 @@ export function registerAcpCli(program: Command) { "after", () => `\n${theme.muted("Docs:")} ${formatDocsLink("/cli/acp", "docs.openclaw.ai/cli/acp")}\n`, ) - .action((opts) => { + .action(async (opts) => { try { - serveAcpGateway({ + await serveAcpGateway({ gatewayUrl: opts.url as string | undefined, gatewayToken: opts.token as string | undefined, gatewayPassword: opts.password as string | undefined, diff --git a/src/cli/argv.test.ts b/src/cli/argv.test.ts index 207a28caefe..a43c6d2e2b7 100644 --- a/src/cli/argv.test.ts +++ b/src/cli/argv.test.ts @@ -144,6 +144,10 @@ describe("argv helpers", () => { expect(shouldMigrateState(["node", "openclaw", "status"])).toBe(false); expect(shouldMigrateState(["node", "openclaw", "health"])).toBe(false); expect(shouldMigrateState(["node", "openclaw", "sessions"])).toBe(false); + expect(shouldMigrateState(["node", "openclaw", "config", "get", "update"])).toBe(false); + expect(shouldMigrateState(["node", "openclaw", "config", "unset", "update"])).toBe(false); + expect(shouldMigrateState(["node", "openclaw", "models", "list"])).toBe(false); + expect(shouldMigrateState(["node", "openclaw", "models", "status"])).toBe(false); expect(shouldMigrateState(["node", "openclaw", "memory", "status"])).toBe(false); expect(shouldMigrateState(["node", "openclaw", "agent", "--message", "hi"])).toBe(false); expect(shouldMigrateState(["node", "openclaw", "agents", "list"])).toBe(true); @@ -152,6 +156,8 @@ describe("argv helpers", () => { it("reuses command path for migrate state decisions", () => { expect(shouldMigrateStateFromPath(["status"])).toBe(false); + expect(shouldMigrateStateFromPath(["config", "get"])).toBe(false); + expect(shouldMigrateStateFromPath(["models", "status"])).toBe(false); expect(shouldMigrateStateFromPath(["agents", "list"])).toBe(true); }); }); diff --git a/src/cli/argv.ts b/src/cli/argv.ts index d922e786383..1489cec4fc3 100644 --- a/src/cli/argv.ts +++ b/src/cli/argv.ts @@ -155,6 +155,12 @@ export function shouldMigrateStateFromPath(path: string[]): boolean { if (primary === "health" || primary === "status" || primary === "sessions") { return false; } + if (primary === "config" && (secondary === "get" || secondary === "unset")) { + return false; + } + if (primary === "models" && (secondary === "list" || secondary === "status")) { + return false; + } if (primary === "memory" && secondary === "status") { return false; } diff --git a/src/cli/completion-cli.ts b/src/cli/completion-cli.ts index 1a65595a765..6874611f89c 100644 --- a/src/cli/completion-cli.ts +++ b/src/cli/completion-cli.ts @@ -3,6 +3,7 @@ import fs from "node:fs/promises"; import os from "node:os"; import path from "node:path"; import { resolveStateDir } from "../config/paths.js"; +import { routeLogsToStderr } from "../logging/console.js"; import { pathExists } from "../utils.js"; import { getSubCliEntries, registerSubCliByName } from "./program/register.subclis.js"; @@ -235,6 +236,9 @@ export function registerCompletionCli(program: Command) { ) .option("-y, --yes", "Skip confirmation (non-interactive)", false) .action(async (options) => { + // Route logs to stderr so plugin loading messages do not corrupt + // the completion script written to stdout. + routeLogsToStderr(); const shell = options.shell ?? "zsh"; // Eagerly register all subcommands to build the full tree const entries = getSubCliEntries(); @@ -269,7 +273,7 @@ export function registerCompletionCli(program: Command) { throw new Error(`Unsupported shell: ${shell}`); } const script = getCompletionScript(shell, program); - console.log(script); + process.stdout.write(script + "\n"); }); } diff --git a/src/cli/config-cli.ts b/src/cli/config-cli.ts index e87ce7c1533..af481916634 100644 --- a/src/cli/config-cli.ts +++ b/src/cli/config-cli.ts @@ -1,5 +1,6 @@ import type { Command } from "commander"; import JSON5 from "json5"; +import type { RuntimeEnv } from "../runtime.js"; import { readConfigFileSnapshot, writeConfigFile } from "../config/config.js"; import { danger, info } from "../globals.js"; import { defaultRuntime } from "../runtime.js"; @@ -201,20 +202,81 @@ function unsetAtPath(root: Record, path: PathSegment[]): boolea return true; } -async function loadValidConfig() { +async function loadValidConfig(runtime: RuntimeEnv = defaultRuntime) { const snapshot = await readConfigFileSnapshot(); if (snapshot.valid) { return snapshot; } - defaultRuntime.error(`Config invalid at ${shortenHomePath(snapshot.path)}.`); + runtime.error(`Config invalid at ${shortenHomePath(snapshot.path)}.`); for (const issue of snapshot.issues) { - defaultRuntime.error(`- ${issue.path || ""}: ${issue.message}`); + runtime.error(`- ${issue.path || ""}: ${issue.message}`); } - defaultRuntime.error(`Run \`${formatCliCommand("openclaw doctor")}\` to repair, then retry.`); - defaultRuntime.exit(1); + runtime.error(`Run \`${formatCliCommand("openclaw doctor")}\` to repair, then retry.`); + runtime.exit(1); return snapshot; } +function parseRequiredPath(path: string): PathSegment[] { + const parsedPath = parsePath(path); + if (parsedPath.length === 0) { + throw new Error("Path is empty."); + } + return parsedPath; +} + +export async function runConfigGet(opts: { path: string; json?: boolean; runtime?: RuntimeEnv }) { + const runtime = opts.runtime ?? defaultRuntime; + try { + const parsedPath = parseRequiredPath(opts.path); + const snapshot = await loadValidConfig(runtime); + const res = getAtPath(snapshot.config, parsedPath); + if (!res.found) { + runtime.error(danger(`Config path not found: ${opts.path}`)); + runtime.exit(1); + return; + } + if (opts.json) { + runtime.log(JSON.stringify(res.value ?? null, null, 2)); + return; + } + if ( + typeof res.value === "string" || + typeof res.value === "number" || + typeof res.value === "boolean" + ) { + runtime.log(String(res.value)); + return; + } + runtime.log(JSON.stringify(res.value ?? null, null, 2)); + } catch (err) { + runtime.error(danger(String(err))); + runtime.exit(1); + } +} + +export async function runConfigUnset(opts: { path: string; runtime?: RuntimeEnv }) { + const runtime = opts.runtime ?? defaultRuntime; + try { + const parsedPath = parseRequiredPath(opts.path); + const snapshot = await loadValidConfig(runtime); + // Use snapshot.resolved (config after $include and ${ENV} resolution, but BEFORE runtime defaults) + // instead of snapshot.config (runtime-merged with defaults). + // This prevents runtime defaults from leaking into the written config file (issue #6070) + const next = structuredClone(snapshot.resolved) as Record; + const removed = unsetAtPath(next, parsedPath); + if (!removed) { + runtime.error(danger(`Config path not found: ${opts.path}`)); + runtime.exit(1); + return; + } + await writeConfigFile(next); + runtime.log(info(`Removed ${opts.path}. Restart the gateway to apply.`)); + } catch (err) { + runtime.error(danger(String(err))); + runtime.exit(1); + } +} + export function registerConfigCli(program: Command) { const cmd = program .command("config") @@ -261,35 +323,7 @@ export function registerConfigCli(program: Command) { .argument("", "Config path (dot or bracket notation)") .option("--json", "Output JSON", false) .action(async (path: string, opts) => { - try { - const parsedPath = parsePath(path); - if (parsedPath.length === 0) { - throw new Error("Path is empty."); - } - const snapshot = await loadValidConfig(); - const res = getAtPath(snapshot.config, parsedPath); - if (!res.found) { - defaultRuntime.error(danger(`Config path not found: ${path}`)); - defaultRuntime.exit(1); - return; - } - if (opts.json) { - defaultRuntime.log(JSON.stringify(res.value ?? null, null, 2)); - return; - } - if ( - typeof res.value === "string" || - typeof res.value === "number" || - typeof res.value === "boolean" - ) { - defaultRuntime.log(String(res.value)); - return; - } - defaultRuntime.log(JSON.stringify(res.value ?? null, null, 2)); - } catch (err) { - defaultRuntime.error(danger(String(err))); - defaultRuntime.exit(1); - } + await runConfigGet({ path, json: Boolean(opts.json) }); }); cmd @@ -324,27 +358,6 @@ export function registerConfigCli(program: Command) { .description("Remove a config value by dot path") .argument("", "Config path (dot or bracket notation)") .action(async (path: string) => { - try { - const parsedPath = parsePath(path); - if (parsedPath.length === 0) { - throw new Error("Path is empty."); - } - const snapshot = await loadValidConfig(); - // Use snapshot.resolved (config after $include and ${ENV} resolution, but BEFORE runtime defaults) - // instead of snapshot.config (runtime-merged with defaults). - // This prevents runtime defaults from leaking into the written config file (issue #6070) - const next = structuredClone(snapshot.resolved) as Record; - const removed = unsetAtPath(next, parsedPath); - if (!removed) { - defaultRuntime.error(danger(`Config path not found: ${path}`)); - defaultRuntime.exit(1); - return; - } - await writeConfigFile(next); - defaultRuntime.log(info(`Removed ${path}. Restart the gateway to apply.`)); - } catch (err) { - defaultRuntime.error(danger(String(err))); - defaultRuntime.exit(1); - } + await runConfigUnset({ path }); }); } diff --git a/src/cli/cron-cli.test.ts b/src/cli/cron-cli.test.ts index 164b951b538..2bd437fb092 100644 --- a/src/cli/cron-cli.test.ts +++ b/src/cli/cron-cli.test.ts @@ -27,14 +27,20 @@ vi.mock("../runtime.js", () => ({ }, })); +const { registerCronCli } = await import("./cron-cli.js"); + +function buildProgram() { + const program = new Command(); + program.exitOverride(); + registerCronCli(program); + return program; +} + describe("cron cli", () => { it("trims model and thinking on cron add", { timeout: 60_000 }, async () => { callGatewayFromCli.mockClear(); - const { registerCronCli } = await import("./cron-cli.js"); - const program = new Command(); - program.exitOverride(); - registerCronCli(program); + const program = buildProgram(); await program.parseAsync( [ @@ -68,10 +74,7 @@ describe("cron cli", () => { it("defaults isolated cron add to announce delivery", async () => { callGatewayFromCli.mockClear(); - const { registerCronCli } = await import("./cron-cli.js"); - const program = new Command(); - program.exitOverride(); - registerCronCli(program); + const program = buildProgram(); await program.parseAsync( [ @@ -98,10 +101,7 @@ describe("cron cli", () => { it("infers sessionTarget from payload when --session is omitted", async () => { callGatewayFromCli.mockClear(); - const { registerCronCli } = await import("./cron-cli.js"); - const program = new Command(); - program.exitOverride(); - registerCronCli(program); + const program = buildProgram(); await program.parseAsync( ["cron", "add", "--name", "Main reminder", "--cron", "* * * * *", "--system-event", "hi"], @@ -129,10 +129,7 @@ describe("cron cli", () => { it("supports --keep-after-run on cron add", async () => { callGatewayFromCli.mockClear(); - const { registerCronCli } = await import("./cron-cli.js"); - const program = new Command(); - program.exitOverride(); - registerCronCli(program); + const program = buildProgram(); await program.parseAsync( [ @@ -159,10 +156,7 @@ describe("cron cli", () => { it("sends agent id on cron add", async () => { callGatewayFromCli.mockClear(); - const { registerCronCli } = await import("./cron-cli.js"); - const program = new Command(); - program.exitOverride(); - registerCronCli(program); + const program = buildProgram(); await program.parseAsync( [ @@ -190,10 +184,7 @@ describe("cron cli", () => { it("omits empty model and thinking on cron edit", async () => { callGatewayFromCli.mockClear(); - const { registerCronCli } = await import("./cron-cli.js"); - const program = new Command(); - program.exitOverride(); - registerCronCli(program); + const program = buildProgram(); await program.parseAsync( ["cron", "edit", "job-1", "--message", "hello", "--model", " ", "--thinking", " "], @@ -212,10 +203,7 @@ describe("cron cli", () => { it("trims model and thinking on cron edit", async () => { callGatewayFromCli.mockClear(); - const { registerCronCli } = await import("./cron-cli.js"); - const program = new Command(); - program.exitOverride(); - registerCronCli(program); + const program = buildProgram(); await program.parseAsync( [ @@ -244,10 +232,7 @@ describe("cron cli", () => { it("sets and clears agent id on cron edit", async () => { callGatewayFromCli.mockClear(); - const { registerCronCli } = await import("./cron-cli.js"); - const program = new Command(); - program.exitOverride(); - registerCronCli(program); + const program = buildProgram(); await program.parseAsync(["cron", "edit", "job-1", "--agent", " Ops ", "--message", "hello"], { from: "user", @@ -269,10 +254,7 @@ describe("cron cli", () => { it("allows model/thinking updates without --message", async () => { callGatewayFromCli.mockClear(); - const { registerCronCli } = await import("./cron-cli.js"); - const program = new Command(); - program.exitOverride(); - registerCronCli(program); + const program = buildProgram(); await program.parseAsync(["cron", "edit", "job-1", "--model", "opus", "--thinking", "low"], { from: "user", @@ -291,10 +273,7 @@ describe("cron cli", () => { it("updates delivery settings without requiring --message", async () => { callGatewayFromCli.mockClear(); - const { registerCronCli } = await import("./cron-cli.js"); - const program = new Command(); - program.exitOverride(); - registerCronCli(program); + const program = buildProgram(); await program.parseAsync( ["cron", "edit", "job-1", "--deliver", "--channel", "telegram", "--to", "19098680"], @@ -319,10 +298,7 @@ describe("cron cli", () => { it("supports --no-deliver on cron edit", async () => { callGatewayFromCli.mockClear(); - const { registerCronCli } = await import("./cron-cli.js"); - const program = new Command(); - program.exitOverride(); - registerCronCli(program); + const program = buildProgram(); await program.parseAsync(["cron", "edit", "job-1", "--no-deliver"], { from: "user" }); @@ -338,10 +314,7 @@ describe("cron cli", () => { it("does not include undefined delivery fields when updating message", async () => { callGatewayFromCli.mockClear(); - const { registerCronCli } = await import("./cron-cli.js"); - const program = new Command(); - program.exitOverride(); - registerCronCli(program); + const program = buildProgram(); // Update message without delivery flags - should NOT include undefined delivery fields await program.parseAsync(["cron", "edit", "job-1", "--message", "Updated message"], { @@ -376,10 +349,7 @@ describe("cron cli", () => { it("includes delivery fields when explicitly provided with message", async () => { callGatewayFromCli.mockClear(); - const { registerCronCli } = await import("./cron-cli.js"); - const program = new Command(); - program.exitOverride(); - registerCronCli(program); + const program = buildProgram(); // Update message AND delivery - should include both await program.parseAsync( @@ -416,10 +386,7 @@ describe("cron cli", () => { it("includes best-effort delivery when provided with message", async () => { callGatewayFromCli.mockClear(); - const { registerCronCli } = await import("./cron-cli.js"); - const program = new Command(); - program.exitOverride(); - registerCronCli(program); + const program = buildProgram(); await program.parseAsync( ["cron", "edit", "job-1", "--message", "Updated message", "--best-effort-deliver"], @@ -442,10 +409,7 @@ describe("cron cli", () => { it("includes no-best-effort delivery when provided with message", async () => { callGatewayFromCli.mockClear(); - const { registerCronCli } = await import("./cron-cli.js"); - const program = new Command(); - program.exitOverride(); - registerCronCli(program); + const program = buildProgram(); await program.parseAsync( ["cron", "edit", "job-1", "--message", "Updated message", "--no-best-effort-deliver"], diff --git a/src/cli/daemon-cli/register.ts b/src/cli/daemon-cli/register.ts index d1599a206aa..47e3dd09bdf 100644 --- a/src/cli/daemon-cli/register.ts +++ b/src/cli/daemon-cli/register.ts @@ -1,7 +1,6 @@ import type { Command } from "commander"; import { formatDocsLink } from "../../terminal/links.js"; import { theme } from "../../terminal/theme.js"; -import { createDefaultDeps } from "../deps.js"; import { runDaemonInstall, runDaemonRestart, @@ -83,7 +82,4 @@ export function registerDaemonCli(program: Command) { .action(async (opts) => { await runDaemonRestart(opts); }); - - // Build default deps (parity with other commands). - void createDefaultDeps(); } diff --git a/src/cli/deps.test.ts b/src/cli/deps.test.ts new file mode 100644 index 00000000000..34c28cece57 --- /dev/null +++ b/src/cli/deps.test.ts @@ -0,0 +1,93 @@ +import { beforeEach, describe, expect, it, vi } from "vitest"; +import { createDefaultDeps } from "./deps.js"; + +const moduleLoads = vi.hoisted(() => ({ + whatsapp: vi.fn(), + telegram: vi.fn(), + discord: vi.fn(), + slack: vi.fn(), + signal: vi.fn(), + imessage: vi.fn(), +})); + +const sendFns = vi.hoisted(() => ({ + whatsapp: vi.fn(async () => ({ messageId: "w1", toJid: "whatsapp:1" })), + telegram: vi.fn(async () => ({ messageId: "t1", chatId: "telegram:1" })), + discord: vi.fn(async () => ({ messageId: "d1", channelId: "discord:1" })), + slack: vi.fn(async () => ({ messageId: "s1", channelId: "slack:1" })), + signal: vi.fn(async () => ({ messageId: "sg1", conversationId: "signal:1" })), + imessage: vi.fn(async () => ({ messageId: "i1", chatId: "imessage:1" })), +})); + +vi.mock("../channels/web/index.js", () => { + moduleLoads.whatsapp(); + return { sendMessageWhatsApp: sendFns.whatsapp }; +}); + +vi.mock("../telegram/send.js", () => { + moduleLoads.telegram(); + return { sendMessageTelegram: sendFns.telegram }; +}); + +vi.mock("../discord/send.js", () => { + moduleLoads.discord(); + return { sendMessageDiscord: sendFns.discord }; +}); + +vi.mock("../slack/send.js", () => { + moduleLoads.slack(); + return { sendMessageSlack: sendFns.slack }; +}); + +vi.mock("../signal/send.js", () => { + moduleLoads.signal(); + return { sendMessageSignal: sendFns.signal }; +}); + +vi.mock("../imessage/send.js", () => { + moduleLoads.imessage(); + return { sendMessageIMessage: sendFns.imessage }; +}); + +describe("createDefaultDeps", () => { + beforeEach(() => { + vi.clearAllMocks(); + }); + + it("does not load provider modules until a dependency is used", async () => { + const deps = createDefaultDeps(); + + expect(moduleLoads.whatsapp).not.toHaveBeenCalled(); + expect(moduleLoads.telegram).not.toHaveBeenCalled(); + expect(moduleLoads.discord).not.toHaveBeenCalled(); + expect(moduleLoads.slack).not.toHaveBeenCalled(); + expect(moduleLoads.signal).not.toHaveBeenCalled(); + expect(moduleLoads.imessage).not.toHaveBeenCalled(); + + const sendTelegram = deps.sendMessageTelegram as unknown as ( + ...args: unknown[] + ) => Promise; + await sendTelegram("chat", "hello", { verbose: false }); + + expect(moduleLoads.telegram).toHaveBeenCalledTimes(1); + expect(sendFns.telegram).toHaveBeenCalledTimes(1); + expect(moduleLoads.whatsapp).not.toHaveBeenCalled(); + expect(moduleLoads.discord).not.toHaveBeenCalled(); + expect(moduleLoads.slack).not.toHaveBeenCalled(); + expect(moduleLoads.signal).not.toHaveBeenCalled(); + expect(moduleLoads.imessage).not.toHaveBeenCalled(); + }); + + it("reuses module cache after first dynamic import", async () => { + const deps = createDefaultDeps(); + const sendDiscord = deps.sendMessageDiscord as unknown as ( + ...args: unknown[] + ) => Promise; + + await sendDiscord("channel", "first", { verbose: false }); + await sendDiscord("channel", "second", { verbose: false }); + + expect(moduleLoads.discord).toHaveBeenCalledTimes(1); + expect(sendFns.discord).toHaveBeenCalledTimes(2); + }); +}); diff --git a/src/cli/deps.ts b/src/cli/deps.ts index 1f0e8e587f0..a3c3c72ac49 100644 --- a/src/cli/deps.ts +++ b/src/cli/deps.ts @@ -1,10 +1,10 @@ +import type { sendMessageWhatsApp } from "../channels/web/index.js"; +import type { sendMessageDiscord } from "../discord/send.js"; +import type { sendMessageIMessage } from "../imessage/send.js"; import type { OutboundSendDeps } from "../infra/outbound/deliver.js"; -import { logWebSelfId, sendMessageWhatsApp } from "../channels/web/index.js"; -import { sendMessageDiscord } from "../discord/send.js"; -import { sendMessageIMessage } from "../imessage/send.js"; -import { sendMessageSignal } from "../signal/send.js"; -import { sendMessageSlack } from "../slack/send.js"; -import { sendMessageTelegram } from "../telegram/send.js"; +import type { sendMessageSignal } from "../signal/send.js"; +import type { sendMessageSlack } from "../slack/send.js"; +import type { sendMessageTelegram } from "../telegram/send.js"; export type CliDeps = { sendMessageWhatsApp: typeof sendMessageWhatsApp; @@ -17,12 +17,30 @@ export type CliDeps = { export function createDefaultDeps(): CliDeps { return { - sendMessageWhatsApp, - sendMessageTelegram, - sendMessageDiscord, - sendMessageSlack, - sendMessageSignal, - sendMessageIMessage, + sendMessageWhatsApp: async (...args) => { + const { sendMessageWhatsApp } = await import("../channels/web/index.js"); + return await sendMessageWhatsApp(...args); + }, + sendMessageTelegram: async (...args) => { + const { sendMessageTelegram } = await import("../telegram/send.js"); + return await sendMessageTelegram(...args); + }, + sendMessageDiscord: async (...args) => { + const { sendMessageDiscord } = await import("../discord/send.js"); + return await sendMessageDiscord(...args); + }, + sendMessageSlack: async (...args) => { + const { sendMessageSlack } = await import("../slack/send.js"); + return await sendMessageSlack(...args); + }, + sendMessageSignal: async (...args) => { + const { sendMessageSignal } = await import("../signal/send.js"); + return await sendMessageSignal(...args); + }, + sendMessageIMessage: async (...args) => { + const { sendMessageIMessage } = await import("../imessage/send.js"); + return await sendMessageIMessage(...args); + }, }; } @@ -38,4 +56,4 @@ export function createOutboundSendDeps(deps: CliDeps): OutboundSendDeps { }; } -export { logWebSelfId }; +export { logWebSelfId } from "../web/auth-store.js"; diff --git a/src/cli/exec-approvals-cli.test.ts b/src/cli/exec-approvals-cli.test.ts index 1d8a1d58dcd..a875d58782c 100644 --- a/src/cli/exec-approvals-cli.test.ts +++ b/src/cli/exec-approvals-cli.test.ts @@ -59,9 +59,11 @@ vi.mock("../infra/exec-approvals.js", async () => { }; }); +const { registerExecApprovalsCli } = await import("./exec-approvals-cli.js"); +const execApprovals = await import("../infra/exec-approvals.js"); + describe("exec approvals CLI", () => { - const createProgram = async () => { - const { registerExecApprovalsCli } = await import("./exec-approvals-cli.js"); + const createProgram = () => { const program = new Command(); program.exitOverride(); registerExecApprovalsCli(program); @@ -73,21 +75,21 @@ describe("exec approvals CLI", () => { runtimeErrors.length = 0; callGatewayFromCli.mockClear(); - const localProgram = await createProgram(); + const localProgram = createProgram(); await localProgram.parseAsync(["approvals", "get"], { from: "user" }); expect(callGatewayFromCli).not.toHaveBeenCalled(); expect(runtimeErrors).toHaveLength(0); callGatewayFromCli.mockClear(); - const gatewayProgram = await createProgram(); + const gatewayProgram = createProgram(); await gatewayProgram.parseAsync(["approvals", "get", "--gateway"], { from: "user" }); expect(callGatewayFromCli).toHaveBeenCalledWith("exec.approvals.get", expect.anything(), {}); expect(runtimeErrors).toHaveLength(0); callGatewayFromCli.mockClear(); - const nodeProgram = await createProgram(); + const nodeProgram = createProgram(); await nodeProgram.parseAsync(["approvals", "get", "--node", "macbook"], { from: "user" }); expect(callGatewayFromCli).toHaveBeenCalledWith("exec.approvals.node.get", expect.anything(), { @@ -101,11 +103,9 @@ describe("exec approvals CLI", () => { runtimeErrors.length = 0; callGatewayFromCli.mockClear(); - const execApprovals = await import("../infra/exec-approvals.js"); const saveExecApprovals = vi.mocked(execApprovals.saveExecApprovals); saveExecApprovals.mockClear(); - const { registerExecApprovalsCli } = await import("./exec-approvals-cli.js"); const program = new Command(); program.exitOverride(); registerExecApprovalsCli(program); diff --git a/src/cli/gateway-cli/run-loop.test.ts b/src/cli/gateway-cli/run-loop.test.ts new file mode 100644 index 00000000000..f2de12bcb57 --- /dev/null +++ b/src/cli/gateway-cli/run-loop.test.ts @@ -0,0 +1,123 @@ +import { describe, expect, it, vi } from "vitest"; + +const acquireGatewayLock = vi.fn(async () => ({ + release: vi.fn(async () => {}), +})); +const consumeGatewaySigusr1RestartAuthorization = vi.fn(() => true); +const isGatewaySigusr1RestartExternallyAllowed = vi.fn(() => false); +const markGatewaySigusr1RestartHandled = vi.fn(); +const getActiveTaskCount = vi.fn(() => 0); +const waitForActiveTasks = vi.fn(async () => ({ drained: true })); +const resetAllLanes = vi.fn(); +const DRAIN_TIMEOUT_LOG = "drain timeout reached; proceeding with restart"; +const gatewayLog = { + info: vi.fn(), + warn: vi.fn(), + error: vi.fn(), +}; + +vi.mock("../../infra/gateway-lock.js", () => ({ + acquireGatewayLock: () => acquireGatewayLock(), +})); + +vi.mock("../../infra/restart.js", () => ({ + consumeGatewaySigusr1RestartAuthorization: () => consumeGatewaySigusr1RestartAuthorization(), + isGatewaySigusr1RestartExternallyAllowed: () => isGatewaySigusr1RestartExternallyAllowed(), + markGatewaySigusr1RestartHandled: () => markGatewaySigusr1RestartHandled(), +})); + +vi.mock("../../process/command-queue.js", () => ({ + getActiveTaskCount: () => getActiveTaskCount(), + waitForActiveTasks: (timeoutMs: number) => waitForActiveTasks(timeoutMs), + resetAllLanes: () => resetAllLanes(), +})); + +vi.mock("../../logging/subsystem.js", () => ({ + createSubsystemLogger: () => gatewayLog, +})); + +function removeNewSignalListeners( + signal: NodeJS.Signals, + existing: Set<(...args: unknown[]) => void>, +) { + for (const listener of process.listeners(signal)) { + const fn = listener as (...args: unknown[]) => void; + if (!existing.has(fn)) { + process.removeListener(signal, fn); + } + } +} + +describe("runGatewayLoop", () => { + it("restarts after SIGUSR1 even when drain times out, and resets lanes for the new iteration", async () => { + vi.clearAllMocks(); + getActiveTaskCount.mockReturnValueOnce(2).mockReturnValueOnce(0); + waitForActiveTasks.mockResolvedValueOnce({ drained: false }); + + type StartServer = () => Promise<{ + close: (opts: { reason: string; restartExpectedMs: number | null }) => Promise; + }>; + + const closeFirst = vi.fn(async () => {}); + const closeSecond = vi.fn(async () => {}); + const start = vi + .fn() + .mockResolvedValueOnce({ close: closeFirst }) + .mockResolvedValueOnce({ close: closeSecond }) + .mockRejectedValueOnce(new Error("stop-loop")); + + const beforeSigterm = new Set( + process.listeners("SIGTERM") as Array<(...args: unknown[]) => void>, + ); + const beforeSigint = new Set( + process.listeners("SIGINT") as Array<(...args: unknown[]) => void>, + ); + const beforeSigusr1 = new Set( + process.listeners("SIGUSR1") as Array<(...args: unknown[]) => void>, + ); + + const loopPromise = import("./run-loop.js").then(({ runGatewayLoop }) => + runGatewayLoop({ + start, + runtime: { + exit: vi.fn(), + } as { exit: (code: number) => never }, + }), + ); + + try { + await vi.waitFor(() => { + expect(start).toHaveBeenCalledTimes(1); + }); + + process.emit("SIGUSR1"); + + await vi.waitFor(() => { + expect(start).toHaveBeenCalledTimes(2); + }); + + expect(waitForActiveTasks).toHaveBeenCalledWith(30_000); + expect(gatewayLog.warn).toHaveBeenCalledWith(DRAIN_TIMEOUT_LOG); + expect(closeFirst).toHaveBeenCalledWith({ + reason: "gateway restarting", + restartExpectedMs: 1500, + }); + expect(markGatewaySigusr1RestartHandled).toHaveBeenCalledTimes(1); + expect(resetAllLanes).toHaveBeenCalledTimes(1); + + process.emit("SIGUSR1"); + + await expect(loopPromise).rejects.toThrow("stop-loop"); + expect(closeSecond).toHaveBeenCalledWith({ + reason: "gateway restarting", + restartExpectedMs: 1500, + }); + expect(markGatewaySigusr1RestartHandled).toHaveBeenCalledTimes(2); + expect(resetAllLanes).toHaveBeenCalledTimes(2); + } finally { + removeNewSignalListeners("SIGTERM", beforeSigterm); + removeNewSignalListeners("SIGINT", beforeSigint); + removeNewSignalListeners("SIGUSR1", beforeSigusr1); + } + }); +}); diff --git a/src/cli/gateway-cli/run-loop.ts b/src/cli/gateway-cli/run-loop.ts index 9486e199e35..7cd1902f57f 100644 --- a/src/cli/gateway-cli/run-loop.ts +++ b/src/cli/gateway-cli/run-loop.ts @@ -4,9 +4,15 @@ import { acquireGatewayLock } from "../../infra/gateway-lock.js"; import { consumeGatewaySigusr1RestartAuthorization, isGatewaySigusr1RestartExternallyAllowed, + markGatewaySigusr1RestartHandled, } from "../../infra/restart.js"; import { createSubsystemLogger } from "../../logging/subsystem.js"; -import { getActiveTaskCount, waitForActiveTasks } from "../../process/command-queue.js"; +import { + getActiveTaskCount, + resetAllLanes, + waitForActiveTasks, +} from "../../process/command-queue.js"; +import { createRestartIterationHook } from "../../process/restart-recovery.js"; const gatewayLog = createSubsystemLogger("gateway"); @@ -103,6 +109,7 @@ export async function runGatewayLoop(params: { ); return; } + markGatewaySigusr1RestartHandled(); request("restart", "SIGUSR1"); }; @@ -111,10 +118,21 @@ export async function runGatewayLoop(params: { process.on("SIGUSR1", onSigusr1); try { + const onIteration = createRestartIterationHook(() => { + // After an in-process restart (SIGUSR1), reset command-queue lane state. + // Interrupted tasks from the previous lifecycle may have left `active` + // counts elevated (their finally blocks never ran), permanently blocking + // new work from draining. This must happen here β€” at the restart + // coordinator level β€” rather than inside individual subsystem init + // functions, to avoid surprising cross-cutting side effects. + resetAllLanes(); + }); + // Keep process alive; SIGUSR1 triggers an in-process restart (no supervisor required). // SIGTERM/SIGINT still exit after a graceful shutdown. // eslint-disable-next-line no-constant-condition while (true) { + onIteration(); server = await params.start(); await new Promise((resolve) => { restartResolver = resolve; diff --git a/src/cli/gateway-cli/run.ts b/src/cli/gateway-cli/run.ts index e3a30bb52a2..2845197efe5 100644 --- a/src/cli/gateway-cli/run.ts +++ b/src/cli/gateway-cli/run.ts @@ -1,11 +1,13 @@ import type { Command } from "commander"; import fs from "node:fs"; +import path from "node:path"; import type { GatewayAuthMode } from "../../config/config.js"; import type { GatewayWsLogStyle } from "../../gateway/ws-logging.js"; import { CONFIG_PATH, loadConfig, readConfigFileSnapshot, + resolveStateDir, resolveGatewayPort, } from "../../config/config.js"; import { resolveGatewayAuth } from "../../gateway/auth.js"; @@ -160,6 +162,7 @@ async function runGatewayCommand(opts: GatewayRunOpts) { const snapshot = await readConfigFileSnapshot().catch(() => null); const configExists = snapshot?.exists ?? fs.existsSync(CONFIG_PATH); + const configAuditPath = path.join(resolveStateDir(process.env), "logs", "config-audit.jsonl"); const mode = cfg.gateway?.mode; if (!opts.allowUnconfigured && mode !== "local") { if (!configExists) { @@ -170,6 +173,7 @@ async function runGatewayCommand(opts: GatewayRunOpts) { defaultRuntime.error( `Gateway start blocked: set gateway.mode=local (current: ${mode ?? "unset"}) or pass --allow-unconfigured.`, ); + defaultRuntime.error(`Config write audit: ${configAuditPath}`); } defaultRuntime.exit(1); return; diff --git a/src/cli/program/command-registry.ts b/src/cli/program/command-registry.ts index 7d7de6bfb8c..1e03dd702d0 100644 --- a/src/cli/program/command-registry.ts +++ b/src/cli/program/command-registry.ts @@ -1,14 +1,8 @@ import type { Command } from "commander"; import type { ProgramContext } from "./context.js"; -import { agentsListCommand } from "../../commands/agents.js"; -import { healthCommand } from "../../commands/health.js"; -import { sessionsCommand } from "../../commands/sessions.js"; -import { statusCommand } from "../../commands/status.js"; -import { defaultRuntime } from "../../runtime.js"; -import { getFlagValue, getPositiveIntFlagValue, getVerboseFlag, hasFlag } from "../argv.js"; import { registerBrowserCli } from "../browser-cli.js"; import { registerConfigCli } from "../config-cli.js"; -import { registerMemoryCli, runMemoryStatus } from "../memory-cli.js"; +import { registerMemoryCli } from "../memory-cli.js"; import { registerAgentCommands } from "./register.agent.js"; import { registerConfigureCommand } from "./register.configure.js"; import { registerMaintenanceCommands } from "./register.maintenance.js"; @@ -24,92 +18,9 @@ type CommandRegisterParams = { argv: string[]; }; -type RouteSpec = { - match: (path: string[]) => boolean; - loadPlugins?: boolean; - run: (argv: string[]) => Promise; -}; - export type CommandRegistration = { id: string; register: (params: CommandRegisterParams) => void; - routes?: RouteSpec[]; -}; - -const routeHealth: RouteSpec = { - match: (path) => path[0] === "health", - loadPlugins: true, - run: async (argv) => { - const json = hasFlag(argv, "--json"); - const verbose = getVerboseFlag(argv, { includeDebug: true }); - const timeoutMs = getPositiveIntFlagValue(argv, "--timeout"); - if (timeoutMs === null) { - return false; - } - await healthCommand({ json, timeoutMs, verbose }, defaultRuntime); - return true; - }, -}; - -const routeStatus: RouteSpec = { - match: (path) => path[0] === "status", - loadPlugins: true, - run: async (argv) => { - const json = hasFlag(argv, "--json"); - const deep = hasFlag(argv, "--deep"); - const all = hasFlag(argv, "--all"); - const usage = hasFlag(argv, "--usage"); - const verbose = getVerboseFlag(argv, { includeDebug: true }); - const timeoutMs = getPositiveIntFlagValue(argv, "--timeout"); - if (timeoutMs === null) { - return false; - } - await statusCommand({ json, deep, all, usage, timeoutMs, verbose }, defaultRuntime); - return true; - }, -}; - -const routeSessions: RouteSpec = { - match: (path) => path[0] === "sessions", - run: async (argv) => { - const json = hasFlag(argv, "--json"); - const store = getFlagValue(argv, "--store"); - if (store === null) { - return false; - } - const active = getFlagValue(argv, "--active"); - if (active === null) { - return false; - } - await sessionsCommand({ json, store, active }, defaultRuntime); - return true; - }, -}; - -const routeAgentsList: RouteSpec = { - match: (path) => path[0] === "agents" && path[1] === "list", - run: async (argv) => { - const json = hasFlag(argv, "--json"); - const bindings = hasFlag(argv, "--bindings"); - await agentsListCommand({ json, bindings }, defaultRuntime); - return true; - }, -}; - -const routeMemoryStatus: RouteSpec = { - match: (path) => path[0] === "memory" && path[1] === "status", - run: async (argv) => { - const agent = getFlagValue(argv, "--agent"); - if (agent === null) { - return false; - } - const json = hasFlag(argv, "--json"); - const deep = hasFlag(argv, "--deep"); - const index = hasFlag(argv, "--index"); - const verbose = hasFlag(argv, "--verbose"); - await runMemoryStatus({ agent, json, deep, index, verbose }); - return true; - }, }; export const commandRegistry: CommandRegistration[] = [ @@ -140,13 +51,11 @@ export const commandRegistry: CommandRegistration[] = [ { id: "memory", register: ({ program }) => registerMemoryCli(program), - routes: [routeMemoryStatus], }, { id: "agent", register: ({ program, ctx }) => registerAgentCommands(program, { agentChannelOptions: ctx.agentChannelOptions }), - routes: [routeAgentsList], }, { id: "subclis", @@ -155,7 +64,6 @@ export const commandRegistry: CommandRegistration[] = [ { id: "status-health-sessions", register: ({ program }) => registerStatusHealthSessionsCommands(program), - routes: [routeHealth, routeStatus, routeSessions], }, { id: "browser", @@ -172,17 +80,3 @@ export function registerProgramCommands( entry.register({ program, ctx, argv }); } } - -export function findRoutedCommand(path: string[]): RouteSpec | null { - for (const entry of commandRegistry) { - if (!entry.routes) { - continue; - } - for (const route of entry.routes) { - if (route.match(path)) { - return route; - } - } - } - return null; -} diff --git a/src/cli/program/config-guard.test.ts b/src/cli/program/config-guard.test.ts new file mode 100644 index 00000000000..d597f7d192f --- /dev/null +++ b/src/cli/program/config-guard.test.ts @@ -0,0 +1,50 @@ +import { beforeEach, describe, expect, it, vi } from "vitest"; + +const loadAndMaybeMigrateDoctorConfigMock = vi.hoisted(() => vi.fn()); +const readConfigFileSnapshotMock = vi.hoisted(() => vi.fn()); + +vi.mock("../../commands/doctor-config-flow.js", () => ({ + loadAndMaybeMigrateDoctorConfig: loadAndMaybeMigrateDoctorConfigMock, +})); + +vi.mock("../../config/config.js", () => ({ + readConfigFileSnapshot: readConfigFileSnapshotMock, +})); + +function makeSnapshot() { + return { + exists: false, + valid: true, + issues: [], + legacyIssues: [], + path: "/tmp/openclaw.json", + }; +} + +function makeRuntime() { + return { + error: vi.fn(), + exit: vi.fn(), + }; +} + +describe("ensureConfigReady", () => { + beforeEach(() => { + vi.clearAllMocks(); + readConfigFileSnapshotMock.mockResolvedValue(makeSnapshot()); + }); + + it("skips doctor flow for read-only fast path commands", async () => { + vi.resetModules(); + const { ensureConfigReady } = await import("./config-guard.js"); + await ensureConfigReady({ runtime: makeRuntime() as never, commandPath: ["status"] }); + expect(loadAndMaybeMigrateDoctorConfigMock).not.toHaveBeenCalled(); + }); + + it("runs doctor flow for commands that may mutate state", async () => { + vi.resetModules(); + const { ensureConfigReady } = await import("./config-guard.js"); + await ensureConfigReady({ runtime: makeRuntime() as never, commandPath: ["message"] }); + expect(loadAndMaybeMigrateDoctorConfigMock).toHaveBeenCalledTimes(1); + }); +}); diff --git a/src/cli/program/config-guard.ts b/src/cli/program/config-guard.ts index 7737de03ccf..0c15b8f1214 100644 --- a/src/cli/program/config-guard.ts +++ b/src/cli/program/config-guard.ts @@ -3,6 +3,7 @@ import { loadAndMaybeMigrateDoctorConfig } from "../../commands/doctor-config-fl import { readConfigFileSnapshot } from "../../config/config.js"; import { colorize, isRich, theme } from "../../terminal/theme.js"; import { shortenHomePath } from "../../utils.js"; +import { shouldMigrateStateFromPath } from "../argv.js"; import { formatCliCommand } from "../command-format.js"; const ALLOWED_INVALID_COMMANDS = new Set(["doctor", "logs", "health", "help", "status"]); @@ -28,7 +29,8 @@ export async function ensureConfigReady(params: { runtime: RuntimeEnv; commandPath?: string[]; }): Promise { - if (!didRunDoctorConfigFlow) { + const commandPath = params.commandPath ?? []; + if (!didRunDoctorConfigFlow && shouldMigrateStateFromPath(commandPath)) { didRunDoctorConfigFlow = true; await loadAndMaybeMigrateDoctorConfig({ options: { nonInteractive: true }, @@ -37,8 +39,8 @@ export async function ensureConfigReady(params: { } const snapshot = await readConfigFileSnapshot(); - const commandName = params.commandPath?.[0]; - const subcommandName = params.commandPath?.[1]; + const commandName = commandPath[0]; + const subcommandName = commandPath[1]; const allowInvalid = commandName ? ALLOWED_INVALID_COMMANDS.has(commandName) || (commandName === "gateway" && diff --git a/src/cli/program/routes.test.ts b/src/cli/program/routes.test.ts new file mode 100644 index 00000000000..1c910a5ac80 --- /dev/null +++ b/src/cli/program/routes.test.ts @@ -0,0 +1,38 @@ +import { describe, expect, it } from "vitest"; +import { findRoutedCommand } from "./routes.js"; + +describe("program routes", () => { + it("matches status route and preserves plugin loading", () => { + const route = findRoutedCommand(["status"]); + expect(route).not.toBeNull(); + expect(route?.loadPlugins).toBe(true); + }); + + it("returns false when status timeout flag value is missing", async () => { + const route = findRoutedCommand(["status"]); + expect(route).not.toBeNull(); + await expect(route?.run(["node", "openclaw", "status", "--timeout"])).resolves.toBe(false); + }); + + it("returns false for sessions route when --store value is missing", async () => { + const route = findRoutedCommand(["sessions"]); + expect(route).not.toBeNull(); + await expect(route?.run(["node", "openclaw", "sessions", "--store"])).resolves.toBe(false); + }); + + it("does not match unknown routes", () => { + expect(findRoutedCommand(["definitely-not-real"])).toBeNull(); + }); + + it("returns false for config get route when path argument is missing", async () => { + const route = findRoutedCommand(["config", "get"]); + expect(route).not.toBeNull(); + await expect(route?.run(["node", "openclaw", "config", "get", "--json"])).resolves.toBe(false); + }); + + it("returns false for config unset route when path argument is missing", async () => { + const route = findRoutedCommand(["config", "unset"]); + expect(route).not.toBeNull(); + await expect(route?.run(["node", "openclaw", "config", "unset"])).resolves.toBe(false); + }); +}); diff --git a/src/cli/program/routes.ts b/src/cli/program/routes.ts new file mode 100644 index 00000000000..866f35fb559 --- /dev/null +++ b/src/cli/program/routes.ts @@ -0,0 +1,256 @@ +import { defaultRuntime } from "../../runtime.js"; +import { getFlagValue, getPositiveIntFlagValue, getVerboseFlag, hasFlag } from "../argv.js"; + +export type RouteSpec = { + match: (path: string[]) => boolean; + loadPlugins?: boolean; + run: (argv: string[]) => Promise; +}; + +const routeHealth: RouteSpec = { + match: (path) => path[0] === "health", + loadPlugins: true, + run: async (argv) => { + const json = hasFlag(argv, "--json"); + const verbose = getVerboseFlag(argv, { includeDebug: true }); + const timeoutMs = getPositiveIntFlagValue(argv, "--timeout"); + if (timeoutMs === null) { + return false; + } + const { healthCommand } = await import("../../commands/health.js"); + await healthCommand({ json, timeoutMs, verbose }, defaultRuntime); + return true; + }, +}; + +const routeStatus: RouteSpec = { + match: (path) => path[0] === "status", + loadPlugins: true, + run: async (argv) => { + const json = hasFlag(argv, "--json"); + const deep = hasFlag(argv, "--deep"); + const all = hasFlag(argv, "--all"); + const usage = hasFlag(argv, "--usage"); + const verbose = getVerboseFlag(argv, { includeDebug: true }); + const timeoutMs = getPositiveIntFlagValue(argv, "--timeout"); + if (timeoutMs === null) { + return false; + } + const { statusCommand } = await import("../../commands/status.js"); + await statusCommand({ json, deep, all, usage, timeoutMs, verbose }, defaultRuntime); + return true; + }, +}; + +const routeSessions: RouteSpec = { + match: (path) => path[0] === "sessions", + run: async (argv) => { + const json = hasFlag(argv, "--json"); + const store = getFlagValue(argv, "--store"); + if (store === null) { + return false; + } + const active = getFlagValue(argv, "--active"); + if (active === null) { + return false; + } + const { sessionsCommand } = await import("../../commands/sessions.js"); + await sessionsCommand({ json, store, active }, defaultRuntime); + return true; + }, +}; + +const routeAgentsList: RouteSpec = { + match: (path) => path[0] === "agents" && path[1] === "list", + run: async (argv) => { + const json = hasFlag(argv, "--json"); + const bindings = hasFlag(argv, "--bindings"); + const { agentsListCommand } = await import("../../commands/agents.js"); + await agentsListCommand({ json, bindings }, defaultRuntime); + return true; + }, +}; + +const routeMemoryStatus: RouteSpec = { + match: (path) => path[0] === "memory" && path[1] === "status", + run: async (argv) => { + const agent = getFlagValue(argv, "--agent"); + if (agent === null) { + return false; + } + const json = hasFlag(argv, "--json"); + const deep = hasFlag(argv, "--deep"); + const index = hasFlag(argv, "--index"); + const verbose = hasFlag(argv, "--verbose"); + const { runMemoryStatus } = await import("../memory-cli.js"); + await runMemoryStatus({ agent, json, deep, index, verbose }); + return true; + }, +}; + +function getCommandPositionals(argv: string[]): string[] { + const out: string[] = []; + const args = argv.slice(2); + for (const arg of args) { + if (!arg || arg === "--") { + break; + } + if (arg.startsWith("-")) { + continue; + } + out.push(arg); + } + return out; +} + +function getFlagValues(argv: string[], name: string): string[] | null { + const values: string[] = []; + const args = argv.slice(2); + for (let i = 0; i < args.length; i += 1) { + const arg = args[i]; + if (!arg || arg === "--") { + break; + } + if (arg === name) { + const next = args[i + 1]; + if (!next || next === "--" || next.startsWith("-")) { + return null; + } + values.push(next); + i += 1; + continue; + } + if (arg.startsWith(`${name}=`)) { + const value = arg.slice(name.length + 1).trim(); + if (!value) { + return null; + } + values.push(value); + } + } + return values; +} + +const routeConfigGet: RouteSpec = { + match: (path) => path[0] === "config" && path[1] === "get", + run: async (argv) => { + const positionals = getCommandPositionals(argv); + const pathArg = positionals[2]; + if (!pathArg) { + return false; + } + const json = hasFlag(argv, "--json"); + const { runConfigGet } = await import("../config-cli.js"); + await runConfigGet({ path: pathArg, json }); + return true; + }, +}; + +const routeConfigUnset: RouteSpec = { + match: (path) => path[0] === "config" && path[1] === "unset", + run: async (argv) => { + const positionals = getCommandPositionals(argv); + const pathArg = positionals[2]; + if (!pathArg) { + return false; + } + const { runConfigUnset } = await import("../config-cli.js"); + await runConfigUnset({ path: pathArg }); + return true; + }, +}; + +const routeModelsList: RouteSpec = { + match: (path) => path[0] === "models" && path[1] === "list", + run: async (argv) => { + const provider = getFlagValue(argv, "--provider"); + if (provider === null) { + return false; + } + const all = hasFlag(argv, "--all"); + const local = hasFlag(argv, "--local"); + const json = hasFlag(argv, "--json"); + const plain = hasFlag(argv, "--plain"); + const { modelsListCommand } = await import("../../commands/models.js"); + await modelsListCommand({ all, local, provider, json, plain }, defaultRuntime); + return true; + }, +}; + +const routeModelsStatus: RouteSpec = { + match: (path) => path[0] === "models" && path[1] === "status", + run: async (argv) => { + const probeProvider = getFlagValue(argv, "--probe-provider"); + if (probeProvider === null) { + return false; + } + const probeTimeout = getFlagValue(argv, "--probe-timeout"); + if (probeTimeout === null) { + return false; + } + const probeConcurrency = getFlagValue(argv, "--probe-concurrency"); + if (probeConcurrency === null) { + return false; + } + const probeMaxTokens = getFlagValue(argv, "--probe-max-tokens"); + if (probeMaxTokens === null) { + return false; + } + const agent = getFlagValue(argv, "--agent"); + if (agent === null) { + return false; + } + const probeProfileValues = getFlagValues(argv, "--probe-profile"); + if (probeProfileValues === null) { + return false; + } + const probeProfile = + probeProfileValues.length === 0 + ? undefined + : probeProfileValues.length === 1 + ? probeProfileValues[0] + : probeProfileValues; + const json = hasFlag(argv, "--json"); + const plain = hasFlag(argv, "--plain"); + const check = hasFlag(argv, "--check"); + const probe = hasFlag(argv, "--probe"); + const { modelsStatusCommand } = await import("../../commands/models.js"); + await modelsStatusCommand( + { + json, + plain, + check, + probe, + probeProvider, + probeProfile, + probeTimeout, + probeConcurrency, + probeMaxTokens, + agent, + }, + defaultRuntime, + ); + return true; + }, +}; + +const routes: RouteSpec[] = [ + routeHealth, + routeStatus, + routeSessions, + routeAgentsList, + routeMemoryStatus, + routeConfigGet, + routeConfigUnset, + routeModelsList, + routeModelsStatus, +]; + +export function findRoutedCommand(path: string[]): RouteSpec | null { + for (const route of routes) { + if (route.match(path)) { + return route; + } + } + return null; +} diff --git a/src/cli/respawn-policy.test.ts b/src/cli/respawn-policy.test.ts new file mode 100644 index 00000000000..25e026b0a56 --- /dev/null +++ b/src/cli/respawn-policy.test.ts @@ -0,0 +1,13 @@ +import { describe, expect, it } from "vitest"; +import { shouldSkipRespawnForArgv } from "./respawn-policy.js"; + +describe("shouldSkipRespawnForArgv", () => { + it("skips respawn for help/version calls", () => { + expect(shouldSkipRespawnForArgv(["node", "openclaw", "--help"])).toBe(true); + expect(shouldSkipRespawnForArgv(["node", "openclaw", "-V"])).toBe(true); + }); + + it("keeps respawn path for normal commands", () => { + expect(shouldSkipRespawnForArgv(["node", "openclaw", "status"])).toBe(false); + }); +}); diff --git a/src/cli/respawn-policy.ts b/src/cli/respawn-policy.ts new file mode 100644 index 00000000000..d0fe1aa22a9 --- /dev/null +++ b/src/cli/respawn-policy.ts @@ -0,0 +1,5 @@ +import { hasHelpOrVersion } from "./argv.js"; + +export function shouldSkipRespawnForArgv(argv: string[]): boolean { + return hasHelpOrVersion(argv); +} diff --git a/src/cli/route.ts b/src/cli/route.ts index 7a1ddf15ae9..38093e93621 100644 --- a/src/cli/route.ts +++ b/src/cli/route.ts @@ -4,8 +4,8 @@ import { VERSION } from "../version.js"; import { getCommandPath, hasHelpOrVersion } from "./argv.js"; import { emitCliBanner } from "./banner.js"; import { ensurePluginRegistryLoaded } from "./plugin-registry.js"; -import { findRoutedCommand } from "./program/command-registry.js"; import { ensureConfigReady } from "./program/config-guard.js"; +import { findRoutedCommand } from "./program/routes.js"; async function prepareRoutedCommand(params: { argv: string[]; diff --git a/src/cli/run-main.exit.test.ts b/src/cli/run-main.exit.test.ts new file mode 100644 index 00000000000..86d74f09640 --- /dev/null +++ b/src/cli/run-main.exit.test.ts @@ -0,0 +1,49 @@ +import process from "node:process"; +import { beforeEach, describe, expect, it, vi } from "vitest"; + +const tryRouteCliMock = vi.hoisted(() => vi.fn()); +const loadDotEnvMock = vi.hoisted(() => vi.fn()); +const normalizeEnvMock = vi.hoisted(() => vi.fn()); +const ensurePathMock = vi.hoisted(() => vi.fn()); +const assertRuntimeMock = vi.hoisted(() => vi.fn()); + +vi.mock("./route.js", () => ({ + tryRouteCli: tryRouteCliMock, +})); + +vi.mock("../infra/dotenv.js", () => ({ + loadDotEnv: loadDotEnvMock, +})); + +vi.mock("../infra/env.js", () => ({ + normalizeEnv: normalizeEnvMock, +})); + +vi.mock("../infra/path-env.js", () => ({ + ensureOpenClawCliOnPath: ensurePathMock, +})); + +vi.mock("../infra/runtime-guard.js", () => ({ + assertSupportedRuntime: assertRuntimeMock, +})); + +const { runCli } = await import("./run-main.js"); + +describe("runCli exit behavior", () => { + beforeEach(() => { + vi.clearAllMocks(); + }); + + it("does not force process.exit after successful routed command", async () => { + tryRouteCliMock.mockResolvedValueOnce(true); + const exitSpy = vi.spyOn(process, "exit").mockImplementation(((code?: number) => { + throw new Error(`unexpected process.exit(${String(code)})`); + }) as typeof process.exit); + + await runCli(["node", "openclaw", "status"]); + + expect(tryRouteCliMock).toHaveBeenCalledWith(["node", "openclaw", "status"]); + expect(exitSpy).not.toHaveBeenCalled(); + exitSpy.mockRestore(); + }); +}); diff --git a/src/cli/run-main.test.ts b/src/cli/run-main.test.ts index 5013b076cb5..c86071f7d80 100644 --- a/src/cli/run-main.test.ts +++ b/src/cli/run-main.test.ts @@ -1,5 +1,10 @@ import { describe, expect, it } from "vitest"; -import { rewriteUpdateFlagArgv } from "./run-main.js"; +import { + rewriteUpdateFlagArgv, + shouldEnsureCliPath, + shouldRegisterPrimarySubcommand, + shouldSkipPluginCommandRegistration, +} from "./run-main.js"; describe("rewriteUpdateFlagArgv", () => { it("leaves argv unchanged when --update is absent", () => { @@ -34,3 +39,85 @@ describe("rewriteUpdateFlagArgv", () => { ]); }); }); + +describe("shouldRegisterPrimarySubcommand", () => { + it("skips eager primary registration for help/version invocations", () => { + expect(shouldRegisterPrimarySubcommand(["node", "openclaw", "status", "--help"])).toBe(false); + expect(shouldRegisterPrimarySubcommand(["node", "openclaw", "-V"])).toBe(false); + }); + + it("keeps eager primary registration for regular command runs", () => { + expect(shouldRegisterPrimarySubcommand(["node", "openclaw", "status"])).toBe(true); + }); +}); + +describe("shouldSkipPluginCommandRegistration", () => { + it("skips plugin registration for root help/version", () => { + expect( + shouldSkipPluginCommandRegistration({ + argv: ["node", "openclaw", "--help"], + primary: null, + hasBuiltinPrimary: false, + }), + ).toBe(true); + }); + + it("skips plugin registration for builtin subcommand help", () => { + expect( + shouldSkipPluginCommandRegistration({ + argv: ["node", "openclaw", "config", "--help"], + primary: "config", + hasBuiltinPrimary: true, + }), + ).toBe(true); + }); + + it("skips plugin registration for builtin command runs", () => { + expect( + shouldSkipPluginCommandRegistration({ + argv: ["node", "openclaw", "sessions", "--json"], + primary: "sessions", + hasBuiltinPrimary: true, + }), + ).toBe(true); + }); + + it("keeps plugin registration for non-builtin help", () => { + expect( + shouldSkipPluginCommandRegistration({ + argv: ["node", "openclaw", "voicecall", "--help"], + primary: "voicecall", + hasBuiltinPrimary: false, + }), + ).toBe(false); + }); + + it("keeps plugin registration for non-builtin command runs", () => { + expect( + shouldSkipPluginCommandRegistration({ + argv: ["node", "openclaw", "voicecall", "status"], + primary: "voicecall", + hasBuiltinPrimary: false, + }), + ).toBe(false); + }); +}); + +describe("shouldEnsureCliPath", () => { + it("skips path bootstrap for help/version invocations", () => { + expect(shouldEnsureCliPath(["node", "openclaw", "--help"])).toBe(false); + expect(shouldEnsureCliPath(["node", "openclaw", "-V"])).toBe(false); + }); + + it("skips path bootstrap for read-only fast paths", () => { + expect(shouldEnsureCliPath(["node", "openclaw", "status"])).toBe(false); + expect(shouldEnsureCliPath(["node", "openclaw", "sessions", "--json"])).toBe(false); + expect(shouldEnsureCliPath(["node", "openclaw", "config", "get", "update"])).toBe(false); + expect(shouldEnsureCliPath(["node", "openclaw", "models", "status", "--json"])).toBe(false); + }); + + it("keeps path bootstrap for mutating or unknown commands", () => { + expect(shouldEnsureCliPath(["node", "openclaw", "message", "send"])).toBe(true); + expect(shouldEnsureCliPath(["node", "openclaw", "voicecall", "status"])).toBe(true); + }); +}); diff --git a/src/cli/run-main.ts b/src/cli/run-main.ts index bad3f91a21f..d90eda95b7b 100644 --- a/src/cli/run-main.ts +++ b/src/cli/run-main.ts @@ -10,7 +10,7 @@ import { ensureOpenClawCliOnPath } from "../infra/path-env.js"; import { assertSupportedRuntime } from "../infra/runtime-guard.js"; import { installUnhandledRejectionHandler } from "../infra/unhandled-rejections.js"; import { enableConsoleCapture } from "../logging.js"; -import { getPrimaryCommand, hasHelpOrVersion } from "./argv.js"; +import { getCommandPath, getPrimaryCommand, hasHelpOrVersion } from "./argv.js"; import { tryRouteCli } from "./route.js"; export function rewriteUpdateFlagArgv(argv: string[]): string[] { @@ -24,11 +24,51 @@ export function rewriteUpdateFlagArgv(argv: string[]): string[] { return next; } +export function shouldRegisterPrimarySubcommand(argv: string[]): boolean { + return !hasHelpOrVersion(argv); +} + +export function shouldSkipPluginCommandRegistration(params: { + argv: string[]; + primary: string | null; + hasBuiltinPrimary: boolean; +}): boolean { + if (params.hasBuiltinPrimary) { + return true; + } + if (!params.primary) { + return hasHelpOrVersion(params.argv); + } + return false; +} + +export function shouldEnsureCliPath(argv: string[]): boolean { + if (hasHelpOrVersion(argv)) { + return false; + } + const [primary, secondary] = getCommandPath(argv, 2); + if (!primary) { + return true; + } + if (primary === "status" || primary === "health" || primary === "sessions") { + return false; + } + if (primary === "config" && (secondary === "get" || secondary === "unset")) { + return false; + } + if (primary === "models" && (secondary === "list" || secondary === "status")) { + return false; + } + return true; +} + export async function runCli(argv: string[] = process.argv) { const normalizedArgv = stripWindowsNodeExec(argv); loadDotEnv({ quiet: true }); normalizeEnv(); - ensureOpenClawCliOnPath(); + if (shouldEnsureCliPath(normalizedArgv)) { + ensureOpenClawCliOnPath(); + } // Enforce the minimum supported runtime before doing any work. assertSupportedRuntime(); @@ -55,12 +95,18 @@ export async function runCli(argv: string[] = process.argv) { const parseArgv = rewriteUpdateFlagArgv(normalizedArgv); // Register the primary subcommand if one exists (for lazy-loading) const primary = getPrimaryCommand(parseArgv); - if (primary) { + if (primary && shouldRegisterPrimarySubcommand(parseArgv)) { const { registerSubCliByName } = await import("./program/register.subclis.js"); await registerSubCliByName(program, primary); } - const shouldSkipPluginRegistration = !primary && hasHelpOrVersion(parseArgv); + const hasBuiltinPrimary = + primary !== null && program.commands.some((command) => command.name() === primary); + const shouldSkipPluginRegistration = shouldSkipPluginCommandRegistration({ + argv: parseArgv, + primary, + hasBuiltinPrimary, + }); if (!shouldSkipPluginRegistration) { // Register plugin CLI commands before parsing const { registerPluginCliCommands } = await import("../plugins/cli.js"); diff --git a/src/cli/update-cli.test.ts b/src/cli/update-cli.test.ts index 4483790a9ee..aa771741270 100644 --- a/src/cli/update-cli.test.ts +++ b/src/cli/update-cli.test.ts @@ -1,7 +1,7 @@ import fs from "node:fs/promises"; import os from "node:os"; import path from "node:path"; -import { beforeEach, describe, expect, it, vi } from "vitest"; +import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; import type { UpdateRunResult } from "../infra/update-runner.js"; const confirm = vi.fn(); @@ -79,7 +79,35 @@ vi.mock("../runtime.js", () => ({ }, })); +const { runGatewayUpdate } = await import("../infra/update-runner.js"); +const { resolveOpenClawPackageRoot } = await import("../infra/openclaw-root.js"); +const { readConfigFileSnapshot, writeConfigFile } = await import("../config/config.js"); +const { checkUpdateStatus, fetchNpmTagVersion, resolveNpmChannelTag } = + await import("../infra/update-check.js"); +const { runCommandWithTimeout } = await import("../process/exec.js"); +const { runDaemonRestart } = await import("./daemon-cli.js"); +const { defaultRuntime } = await import("../runtime.js"); +const { updateCommand, registerUpdateCli, updateStatusCommand, updateWizardCommand } = + await import("./update-cli.js"); + describe("update-cli", () => { + let fixtureRoot = ""; + let fixtureCount = 0; + + const createCaseDir = async (prefix: string) => { + const dir = path.join(fixtureRoot, `${prefix}-${fixtureCount++}`); + await fs.mkdir(dir, { recursive: true }); + return dir; + }; + + beforeAll(async () => { + fixtureRoot = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-update-tests-")); + }); + + afterAll(async () => { + await fs.rm(fixtureRoot, { recursive: true, force: true }); + }); + const baseSnapshot = { valid: true, config: {}, @@ -100,13 +128,8 @@ describe("update-cli", () => { }); }; - beforeEach(async () => { + beforeEach(() => { vi.clearAllMocks(); - const { resolveOpenClawPackageRoot } = await import("../infra/openclaw-root.js"); - const { readConfigFileSnapshot } = await import("../config/config.js"); - const { checkUpdateStatus, fetchNpmTagVersion, resolveNpmChannelTag } = - await import("../infra/update-check.js"); - const { runCommandWithTimeout } = await import("../process/exec.js"); vi.mocked(resolveOpenClawPackageRoot).mockResolvedValue(process.cwd()); vi.mocked(readConfigFileSnapshot).mockResolvedValue(baseSnapshot); vi.mocked(fetchNpmTagVersion).mockResolvedValue({ @@ -154,18 +177,12 @@ describe("update-cli", () => { }); it("exports updateCommand and registerUpdateCli", async () => { - const { updateCommand, registerUpdateCli, updateWizardCommand } = - await import("./update-cli.js"); expect(typeof updateCommand).toBe("function"); expect(typeof registerUpdateCli).toBe("function"); expect(typeof updateWizardCommand).toBe("function"); }, 20_000); it("updateCommand runs update and outputs result", async () => { - const { runGatewayUpdate } = await import("../infra/update-runner.js"); - const { defaultRuntime } = await import("../runtime.js"); - const { updateCommand } = await import("./update-cli.js"); - const mockResult: UpdateRunResult = { status: "ok", mode: "git", @@ -193,9 +210,6 @@ describe("update-cli", () => { }); it("updateStatusCommand prints table output", async () => { - const { defaultRuntime } = await import("../runtime.js"); - const { updateStatusCommand } = await import("./update-cli.js"); - await updateStatusCommand({ json: false }); const logs = vi.mocked(defaultRuntime.log).mock.calls.map((call) => call[0]); @@ -203,9 +217,6 @@ describe("update-cli", () => { }); it("updateStatusCommand emits JSON", async () => { - const { defaultRuntime } = await import("../runtime.js"); - const { updateStatusCommand } = await import("./update-cli.js"); - await updateStatusCommand({ json: true }); const last = vi.mocked(defaultRuntime.log).mock.calls.at(-1)?.[0]; @@ -215,9 +226,6 @@ describe("update-cli", () => { }); it("defaults to dev channel for git installs when unset", async () => { - const { runGatewayUpdate } = await import("../infra/update-runner.js"); - const { updateCommand } = await import("./update-cli.js"); - vi.mocked(runGatewayUpdate).mockResolvedValue({ status: "ok", mode: "git", @@ -232,53 +240,40 @@ describe("update-cli", () => { }); it("defaults to stable channel for package installs when unset", async () => { - const tempDir = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-update-")); - try { - await fs.writeFile( - path.join(tempDir, "package.json"), - JSON.stringify({ name: "openclaw", version: "1.0.0" }), - "utf-8", - ); + const tempDir = await createCaseDir("openclaw-update"); + await fs.writeFile( + path.join(tempDir, "package.json"), + JSON.stringify({ name: "openclaw", version: "1.0.0" }), + "utf-8", + ); - const { resolveOpenClawPackageRoot } = await import("../infra/openclaw-root.js"); - const { runGatewayUpdate } = await import("../infra/update-runner.js"); - const { checkUpdateStatus } = await import("../infra/update-check.js"); - const { updateCommand } = await import("./update-cli.js"); - - vi.mocked(resolveOpenClawPackageRoot).mockResolvedValue(tempDir); - vi.mocked(checkUpdateStatus).mockResolvedValue({ - root: tempDir, - installKind: "package", - packageManager: "npm", - deps: { - manager: "npm", - status: "ok", - lockfilePath: null, - markerPath: null, - }, - }); - vi.mocked(runGatewayUpdate).mockResolvedValue({ + vi.mocked(resolveOpenClawPackageRoot).mockResolvedValue(tempDir); + vi.mocked(checkUpdateStatus).mockResolvedValue({ + root: tempDir, + installKind: "package", + packageManager: "npm", + deps: { + manager: "npm", status: "ok", - mode: "npm", - steps: [], - durationMs: 100, - }); + lockfilePath: null, + markerPath: null, + }, + }); + vi.mocked(runGatewayUpdate).mockResolvedValue({ + status: "ok", + mode: "npm", + steps: [], + durationMs: 100, + }); - await updateCommand({ yes: true }); + await updateCommand({ yes: true }); - const call = vi.mocked(runGatewayUpdate).mock.calls[0]?.[0]; - expect(call?.channel).toBe("stable"); - expect(call?.tag).toBe("latest"); - } finally { - await fs.rm(tempDir, { recursive: true, force: true }); - } + const call = vi.mocked(runGatewayUpdate).mock.calls[0]?.[0]; + expect(call?.channel).toBe("stable"); + expect(call?.tag).toBe("latest"); }); it("uses stored beta channel when configured", async () => { - const { readConfigFileSnapshot } = await import("../config/config.js"); - const { runGatewayUpdate } = await import("../infra/update-runner.js"); - const { updateCommand } = await import("./update-cli.js"); - vi.mocked(readConfigFileSnapshot).mockResolvedValue({ ...baseSnapshot, config: { update: { channel: "beta" } }, @@ -297,93 +292,70 @@ describe("update-cli", () => { }); it("falls back to latest when beta tag is older than release", async () => { - const tempDir = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-update-")); - try { - await fs.writeFile( - path.join(tempDir, "package.json"), - JSON.stringify({ name: "openclaw", version: "1.0.0" }), - "utf-8", - ); + const tempDir = await createCaseDir("openclaw-update"); + await fs.writeFile( + path.join(tempDir, "package.json"), + JSON.stringify({ name: "openclaw", version: "1.0.0" }), + "utf-8", + ); - const { resolveOpenClawPackageRoot } = await import("../infra/openclaw-root.js"); - const { readConfigFileSnapshot } = await import("../config/config.js"); - const { resolveNpmChannelTag } = await import("../infra/update-check.js"); - const { runGatewayUpdate } = await import("../infra/update-runner.js"); - const { updateCommand } = await import("./update-cli.js"); - const { checkUpdateStatus } = await import("../infra/update-check.js"); - - vi.mocked(resolveOpenClawPackageRoot).mockResolvedValue(tempDir); - vi.mocked(readConfigFileSnapshot).mockResolvedValue({ - ...baseSnapshot, - config: { update: { channel: "beta" } }, - }); - vi.mocked(checkUpdateStatus).mockResolvedValue({ - root: tempDir, - installKind: "package", - packageManager: "npm", - deps: { - manager: "npm", - status: "ok", - lockfilePath: null, - markerPath: null, - }, - }); - vi.mocked(resolveNpmChannelTag).mockResolvedValue({ - tag: "latest", - version: "1.2.3-1", - }); - vi.mocked(runGatewayUpdate).mockResolvedValue({ + vi.mocked(resolveOpenClawPackageRoot).mockResolvedValue(tempDir); + vi.mocked(readConfigFileSnapshot).mockResolvedValue({ + ...baseSnapshot, + config: { update: { channel: "beta" } }, + }); + vi.mocked(checkUpdateStatus).mockResolvedValue({ + root: tempDir, + installKind: "package", + packageManager: "npm", + deps: { + manager: "npm", status: "ok", - mode: "npm", - steps: [], - durationMs: 100, - }); + lockfilePath: null, + markerPath: null, + }, + }); + vi.mocked(resolveNpmChannelTag).mockResolvedValue({ + tag: "latest", + version: "1.2.3-1", + }); + vi.mocked(runGatewayUpdate).mockResolvedValue({ + status: "ok", + mode: "npm", + steps: [], + durationMs: 100, + }); - await updateCommand({}); + await updateCommand({}); - const call = vi.mocked(runGatewayUpdate).mock.calls[0]?.[0]; - expect(call?.channel).toBe("beta"); - expect(call?.tag).toBe("latest"); - } finally { - await fs.rm(tempDir, { recursive: true, force: true }); - } + const call = vi.mocked(runGatewayUpdate).mock.calls[0]?.[0]; + expect(call?.channel).toBe("beta"); + expect(call?.tag).toBe("latest"); }); it("honors --tag override", async () => { - const tempDir = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-update-")); - try { - await fs.writeFile( - path.join(tempDir, "package.json"), - JSON.stringify({ name: "openclaw", version: "1.0.0" }), - "utf-8", - ); + const tempDir = await createCaseDir("openclaw-update"); + await fs.writeFile( + path.join(tempDir, "package.json"), + JSON.stringify({ name: "openclaw", version: "1.0.0" }), + "utf-8", + ); - const { resolveOpenClawPackageRoot } = await import("../infra/openclaw-root.js"); - const { runGatewayUpdate } = await import("../infra/update-runner.js"); - const { updateCommand } = await import("./update-cli.js"); + vi.mocked(resolveOpenClawPackageRoot).mockResolvedValue(tempDir); + vi.mocked(runGatewayUpdate).mockResolvedValue({ + status: "ok", + mode: "npm", + steps: [], + durationMs: 100, + }); - vi.mocked(resolveOpenClawPackageRoot).mockResolvedValue(tempDir); - vi.mocked(runGatewayUpdate).mockResolvedValue({ - status: "ok", - mode: "npm", - steps: [], - durationMs: 100, - }); + await updateCommand({ tag: "next" }); - await updateCommand({ tag: "next" }); - - const call = vi.mocked(runGatewayUpdate).mock.calls[0]?.[0]; - expect(call?.tag).toBe("next"); - } finally { - await fs.rm(tempDir, { recursive: true, force: true }); - } + const call = vi.mocked(runGatewayUpdate).mock.calls[0]?.[0]; + expect(call?.tag).toBe("next"); }); it("updateCommand outputs JSON when --json is set", async () => { - const { runGatewayUpdate } = await import("../infra/update-runner.js"); - const { defaultRuntime } = await import("../runtime.js"); - const { updateCommand } = await import("./update-cli.js"); - const mockResult: UpdateRunResult = { status: "ok", mode: "git", @@ -409,10 +381,6 @@ describe("update-cli", () => { }); it("updateCommand exits with error on failure", async () => { - const { runGatewayUpdate } = await import("../infra/update-runner.js"); - const { defaultRuntime } = await import("../runtime.js"); - const { updateCommand } = await import("./update-cli.js"); - const mockResult: UpdateRunResult = { status: "error", mode: "git", @@ -430,10 +398,6 @@ describe("update-cli", () => { }); it("updateCommand restarts daemon by default", async () => { - const { runGatewayUpdate } = await import("../infra/update-runner.js"); - const { runDaemonRestart } = await import("./daemon-cli.js"); - const { updateCommand } = await import("./update-cli.js"); - const mockResult: UpdateRunResult = { status: "ok", mode: "git", @@ -450,10 +414,6 @@ describe("update-cli", () => { }); it("updateCommand skips restart when --no-restart is set", async () => { - const { runGatewayUpdate } = await import("../infra/update-runner.js"); - const { runDaemonRestart } = await import("./daemon-cli.js"); - const { updateCommand } = await import("./update-cli.js"); - const mockResult: UpdateRunResult = { status: "ok", mode: "git", @@ -469,11 +429,6 @@ describe("update-cli", () => { }); it("updateCommand skips success message when restart does not run", async () => { - const { runGatewayUpdate } = await import("../infra/update-runner.js"); - const { runDaemonRestart } = await import("./daemon-cli.js"); - const { defaultRuntime } = await import("../runtime.js"); - const { updateCommand } = await import("./update-cli.js"); - const mockResult: UpdateRunResult = { status: "ok", mode: "git", @@ -492,9 +447,6 @@ describe("update-cli", () => { }); it("updateCommand validates timeout option", async () => { - const { defaultRuntime } = await import("../runtime.js"); - const { updateCommand } = await import("./update-cli.js"); - vi.mocked(defaultRuntime.error).mockClear(); vi.mocked(defaultRuntime.exit).mockClear(); @@ -505,10 +457,6 @@ describe("update-cli", () => { }); it("persists update channel when --channel is set", async () => { - const { writeConfigFile } = await import("../config/config.js"); - const { runGatewayUpdate } = await import("../infra/update-runner.js"); - const { updateCommand } = await import("./update-cli.js"); - const mockResult: UpdateRunResult = { status: "ok", mode: "git", @@ -528,115 +476,90 @@ describe("update-cli", () => { }); it("requires confirmation on downgrade when non-interactive", async () => { - const tempDir = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-update-")); - try { - setTty(false); - await fs.writeFile( - path.join(tempDir, "package.json"), - JSON.stringify({ name: "openclaw", version: "2.0.0" }), - "utf-8", - ); + const tempDir = await createCaseDir("openclaw-update"); + setTty(false); + await fs.writeFile( + path.join(tempDir, "package.json"), + JSON.stringify({ name: "openclaw", version: "2.0.0" }), + "utf-8", + ); - const { resolveOpenClawPackageRoot } = await import("../infra/openclaw-root.js"); - const { resolveNpmChannelTag } = await import("../infra/update-check.js"); - const { runGatewayUpdate } = await import("../infra/update-runner.js"); - const { defaultRuntime } = await import("../runtime.js"); - const { updateCommand } = await import("./update-cli.js"); - const { checkUpdateStatus } = await import("../infra/update-check.js"); - - vi.mocked(resolveOpenClawPackageRoot).mockResolvedValue(tempDir); - vi.mocked(checkUpdateStatus).mockResolvedValue({ - root: tempDir, - installKind: "package", - packageManager: "npm", - deps: { - manager: "npm", - status: "ok", - lockfilePath: null, - markerPath: null, - }, - }); - vi.mocked(resolveNpmChannelTag).mockResolvedValue({ - tag: "latest", - version: "0.0.1", - }); - vi.mocked(runGatewayUpdate).mockResolvedValue({ + vi.mocked(resolveOpenClawPackageRoot).mockResolvedValue(tempDir); + vi.mocked(checkUpdateStatus).mockResolvedValue({ + root: tempDir, + installKind: "package", + packageManager: "npm", + deps: { + manager: "npm", status: "ok", - mode: "npm", - steps: [], - durationMs: 100, - }); - vi.mocked(defaultRuntime.error).mockClear(); - vi.mocked(defaultRuntime.exit).mockClear(); + lockfilePath: null, + markerPath: null, + }, + }); + vi.mocked(resolveNpmChannelTag).mockResolvedValue({ + tag: "latest", + version: "0.0.1", + }); + vi.mocked(runGatewayUpdate).mockResolvedValue({ + status: "ok", + mode: "npm", + steps: [], + durationMs: 100, + }); + vi.mocked(defaultRuntime.error).mockClear(); + vi.mocked(defaultRuntime.exit).mockClear(); - await updateCommand({}); + await updateCommand({}); - expect(defaultRuntime.error).toHaveBeenCalledWith( - expect.stringContaining("Downgrade confirmation required."), - ); - expect(defaultRuntime.exit).toHaveBeenCalledWith(1); - } finally { - await fs.rm(tempDir, { recursive: true, force: true }); - } + expect(defaultRuntime.error).toHaveBeenCalledWith( + expect.stringContaining("Downgrade confirmation required."), + ); + expect(defaultRuntime.exit).toHaveBeenCalledWith(1); }); it("allows downgrade with --yes in non-interactive mode", async () => { - const tempDir = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-update-")); - try { - setTty(false); - await fs.writeFile( - path.join(tempDir, "package.json"), - JSON.stringify({ name: "openclaw", version: "2.0.0" }), - "utf-8", - ); + const tempDir = await createCaseDir("openclaw-update"); + setTty(false); + await fs.writeFile( + path.join(tempDir, "package.json"), + JSON.stringify({ name: "openclaw", version: "2.0.0" }), + "utf-8", + ); - const { resolveOpenClawPackageRoot } = await import("../infra/openclaw-root.js"); - const { resolveNpmChannelTag } = await import("../infra/update-check.js"); - const { runGatewayUpdate } = await import("../infra/update-runner.js"); - const { defaultRuntime } = await import("../runtime.js"); - const { updateCommand } = await import("./update-cli.js"); - const { checkUpdateStatus } = await import("../infra/update-check.js"); - - vi.mocked(resolveOpenClawPackageRoot).mockResolvedValue(tempDir); - vi.mocked(checkUpdateStatus).mockResolvedValue({ - root: tempDir, - installKind: "package", - packageManager: "npm", - deps: { - manager: "npm", - status: "ok", - lockfilePath: null, - markerPath: null, - }, - }); - vi.mocked(resolveNpmChannelTag).mockResolvedValue({ - tag: "latest", - version: "0.0.1", - }); - vi.mocked(runGatewayUpdate).mockResolvedValue({ + vi.mocked(resolveOpenClawPackageRoot).mockResolvedValue(tempDir); + vi.mocked(checkUpdateStatus).mockResolvedValue({ + root: tempDir, + installKind: "package", + packageManager: "npm", + deps: { + manager: "npm", status: "ok", - mode: "npm", - steps: [], - durationMs: 100, - }); - vi.mocked(defaultRuntime.error).mockClear(); - vi.mocked(defaultRuntime.exit).mockClear(); + lockfilePath: null, + markerPath: null, + }, + }); + vi.mocked(resolveNpmChannelTag).mockResolvedValue({ + tag: "latest", + version: "0.0.1", + }); + vi.mocked(runGatewayUpdate).mockResolvedValue({ + status: "ok", + mode: "npm", + steps: [], + durationMs: 100, + }); + vi.mocked(defaultRuntime.error).mockClear(); + vi.mocked(defaultRuntime.exit).mockClear(); - await updateCommand({ yes: true }); + await updateCommand({ yes: true }); - expect(defaultRuntime.error).not.toHaveBeenCalledWith( - expect.stringContaining("Downgrade confirmation required."), - ); - expect(runGatewayUpdate).toHaveBeenCalled(); - } finally { - await fs.rm(tempDir, { recursive: true, force: true }); - } + expect(defaultRuntime.error).not.toHaveBeenCalledWith( + expect.stringContaining("Downgrade confirmation required."), + ); + expect(runGatewayUpdate).toHaveBeenCalled(); }); it("updateWizardCommand requires a TTY", async () => { - const { defaultRuntime } = await import("../runtime.js"); - const { updateWizardCommand } = await import("./update-cli.js"); - setTty(false); vi.mocked(defaultRuntime.error).mockClear(); vi.mocked(defaultRuntime.exit).mockClear(); @@ -650,16 +573,12 @@ describe("update-cli", () => { }); it("updateWizardCommand offers dev checkout and forwards selections", async () => { - const tempDir = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-update-wizard-")); + const tempDir = await createCaseDir("openclaw-update-wizard"); const previousGitDir = process.env.OPENCLAW_GIT_DIR; try { setTty(true); process.env.OPENCLAW_GIT_DIR = tempDir; - const { checkUpdateStatus } = await import("../infra/update-check.js"); - const { runGatewayUpdate } = await import("../infra/update-runner.js"); - const { updateWizardCommand } = await import("./update-cli.js"); - vi.mocked(checkUpdateStatus).mockResolvedValue({ root: "/test/path", installKind: "package", @@ -686,7 +605,6 @@ describe("update-cli", () => { expect(call?.channel).toBe("dev"); } finally { process.env.OPENCLAW_GIT_DIR = previousGitDir; - await fs.rm(tempDir, { recursive: true, force: true }); } }); }); diff --git a/src/commands/agent/session.test.ts b/src/commands/agent/session.test.ts index 1bae455a26a..93de40b642b 100644 --- a/src/commands/agent/session.test.ts +++ b/src/commands/agent/session.test.ts @@ -22,21 +22,17 @@ vi.mock("../../agents/agent-scope.js", () => ({ listAgentIds: mocks.listAgentIds, })); +const { resolveSessionKeyForRequest } = await import("./session.js"); + describe("resolveSessionKeyForRequest", () => { beforeEach(() => { vi.clearAllMocks(); mocks.listAgentIds.mockReturnValue(["main"]); }); - async function importFresh() { - return await import("./session.js"); - } - const baseCfg: OpenClawConfig = {}; it("returns sessionKey when --to resolves a session key via context", async () => { - const { resolveSessionKeyForRequest } = await importFresh(); - mocks.resolveStorePath.mockReturnValue("/tmp/main-store.json"); mocks.loadSessionStore.mockReturnValue({ "agent:main:main": { sessionId: "sess-1", updatedAt: 0 }, @@ -50,8 +46,6 @@ describe("resolveSessionKeyForRequest", () => { }); it("finds session by sessionId via reverse lookup in primary store", async () => { - const { resolveSessionKeyForRequest } = await importFresh(); - mocks.resolveStorePath.mockReturnValue("/tmp/main-store.json"); mocks.loadSessionStore.mockReturnValue({ "agent:main:main": { sessionId: "target-session-id", updatedAt: 0 }, @@ -65,8 +59,6 @@ describe("resolveSessionKeyForRequest", () => { }); it("finds session by sessionId in non-primary agent store", async () => { - const { resolveSessionKeyForRequest } = await importFresh(); - mocks.listAgentIds.mockReturnValue(["main", "mybot"]); mocks.resolveStorePath.mockImplementation( (_store: string | undefined, opts?: { agentId?: string }) => { @@ -94,8 +86,6 @@ describe("resolveSessionKeyForRequest", () => { }); it("returns correct sessionStore when session found in non-primary agent store", async () => { - const { resolveSessionKeyForRequest } = await importFresh(); - const mybotStore = { "agent:mybot:main": { sessionId: "target-session-id", updatedAt: 0 }, }; @@ -123,8 +113,6 @@ describe("resolveSessionKeyForRequest", () => { }); it("returns undefined sessionKey when sessionId not found in any store", async () => { - const { resolveSessionKeyForRequest } = await importFresh(); - mocks.listAgentIds.mockReturnValue(["main", "mybot"]); mocks.resolveStorePath.mockImplementation( (_store: string | undefined, opts?: { agentId?: string }) => { @@ -144,8 +132,6 @@ describe("resolveSessionKeyForRequest", () => { }); it("does not search other stores when explicitSessionKey is set", async () => { - const { resolveSessionKeyForRequest } = await importFresh(); - mocks.listAgentIds.mockReturnValue(["main", "mybot"]); mocks.resolveStorePath.mockReturnValue("/tmp/main-store.json"); mocks.loadSessionStore.mockReturnValue({ @@ -162,8 +148,6 @@ describe("resolveSessionKeyForRequest", () => { }); it("searches other stores when --to derives a key that does not match --session-id", async () => { - const { resolveSessionKeyForRequest } = await importFresh(); - mocks.listAgentIds.mockReturnValue(["main", "mybot"]); mocks.resolveStorePath.mockImplementation( (_store: string | undefined, opts?: { agentId?: string }) => { @@ -199,8 +183,6 @@ describe("resolveSessionKeyForRequest", () => { }); it("skips already-searched primary store when iterating agents", async () => { - const { resolveSessionKeyForRequest } = await importFresh(); - mocks.listAgentIds.mockReturnValue(["main", "mybot"]); mocks.resolveStorePath.mockImplementation( (_store: string | undefined, opts?: { agentId?: string }) => { diff --git a/src/commands/auth-choice.apply.plugin-provider.ts b/src/commands/auth-choice.apply.plugin-provider.ts index 5555c17266d..5d17f677a56 100644 --- a/src/commands/auth-choice.apply.plugin-provider.ts +++ b/src/commands/auth-choice.apply.plugin-provider.ts @@ -1,5 +1,3 @@ -import type { OpenClawConfig } from "../config/config.js"; -import type { ProviderAuthMethod, ProviderPlugin } from "../plugins/types.js"; import type { ApplyAuthChoiceParams, ApplyAuthChoiceResult } from "./auth-choice.apply.js"; import { resolveOpenClawAgentDir } from "../agents/agent-paths.js"; import { @@ -8,7 +6,6 @@ import { resolveAgentWorkspaceDir, } from "../agents/agent-scope.js"; import { upsertAuthProfile } from "../agents/auth-profiles.js"; -import { normalizeProviderId } from "../agents/model-selection.js"; import { resolveDefaultAgentWorkspaceDir } from "../agents/workspace.js"; import { enablePluginInConfig } from "../plugins/enable.js"; import { resolvePluginProviders } from "../plugins/providers.js"; @@ -16,6 +13,12 @@ import { isRemoteEnvironment } from "./oauth-env.js"; import { createVpsAwareOAuthHandlers } from "./oauth-flow.js"; import { applyAuthProfileConfig } from "./onboard-auth.js"; import { openUrl } from "./onboard-helpers.js"; +import { + applyDefaultModel, + mergeConfigPatch, + pickAuthMethod, + resolveProviderMatch, +} from "./provider-auth-helpers.js"; export type PluginProviderAuthChoiceOptions = { authChoice: string; @@ -25,78 +28,6 @@ export type PluginProviderAuthChoiceOptions = { label: string; }; -function resolveProviderMatch( - providers: ProviderPlugin[], - rawProvider: string, -): ProviderPlugin | null { - const normalized = normalizeProviderId(rawProvider); - return ( - providers.find((provider) => normalizeProviderId(provider.id) === normalized) ?? - providers.find( - (provider) => - provider.aliases?.some((alias) => normalizeProviderId(alias) === normalized) ?? false, - ) ?? - null - ); -} - -function pickAuthMethod(provider: ProviderPlugin, rawMethod?: string): ProviderAuthMethod | null { - const raw = rawMethod?.trim(); - if (!raw) { - return null; - } - const normalized = raw.toLowerCase(); - return ( - provider.auth.find((method) => method.id.toLowerCase() === normalized) ?? - provider.auth.find((method) => method.label.toLowerCase() === normalized) ?? - null - ); -} - -function isPlainRecord(value: unknown): value is Record { - return Boolean(value && typeof value === "object" && !Array.isArray(value)); -} - -function mergeConfigPatch(base: T, patch: unknown): T { - if (!isPlainRecord(base) || !isPlainRecord(patch)) { - return patch as T; - } - - const next: Record = { ...base }; - for (const [key, value] of Object.entries(patch)) { - const existing = next[key]; - if (isPlainRecord(existing) && isPlainRecord(value)) { - next[key] = mergeConfigPatch(existing, value); - } else { - next[key] = value; - } - } - return next as T; -} - -function applyDefaultModel(cfg: OpenClawConfig, model: string): OpenClawConfig { - const models = { ...cfg.agents?.defaults?.models }; - models[model] = models[model] ?? {}; - - const existingModel = cfg.agents?.defaults?.model; - return { - ...cfg, - agents: { - ...cfg.agents, - defaults: { - ...cfg.agents?.defaults, - models, - model: { - ...(existingModel && typeof existingModel === "object" && "fallbacks" in existingModel - ? { fallbacks: (existingModel as { fallbacks?: string[] }).fallbacks } - : undefined), - primary: model, - }, - }, - }, - }; -} - export async function applyAuthChoicePluginProvider( params: ApplyAuthChoiceParams, options: PluginProviderAuthChoiceOptions, diff --git a/src/commands/models.auth.provider-resolution.test.ts b/src/commands/models.auth.provider-resolution.test.ts new file mode 100644 index 00000000000..11cc6d934ac --- /dev/null +++ b/src/commands/models.auth.provider-resolution.test.ts @@ -0,0 +1,48 @@ +import { describe, expect, it } from "vitest"; +import type { ProviderPlugin } from "../plugins/types.js"; +import { resolveRequestedLoginProviderOrThrow } from "./models/auth.js"; + +function makeProvider(params: { id: string; label?: string; aliases?: string[] }): ProviderPlugin { + return { + id: params.id, + label: params.label ?? params.id, + aliases: params.aliases, + auth: [], + }; +} + +describe("resolveRequestedLoginProviderOrThrow", () => { + it("returns null when no provider was requested", () => { + const providers = [makeProvider({ id: "google-antigravity" })]; + const result = resolveRequestedLoginProviderOrThrow(providers, undefined); + expect(result).toBeNull(); + }); + + it("resolves requested provider by id", () => { + const providers = [ + makeProvider({ id: "google-antigravity" }), + makeProvider({ id: "google-gemini-cli" }), + ]; + const result = resolveRequestedLoginProviderOrThrow(providers, "google-antigravity"); + expect(result?.id).toBe("google-antigravity"); + }); + + it("resolves requested provider by alias", () => { + const providers = [makeProvider({ id: "google-antigravity", aliases: ["antigravity"] })]; + const result = resolveRequestedLoginProviderOrThrow(providers, "antigravity"); + expect(result?.id).toBe("google-antigravity"); + }); + + it("throws when requested provider is not loaded", () => { + const providers = [ + makeProvider({ id: "google-gemini-cli" }), + makeProvider({ id: "qwen-portal" }), + ]; + + expect(() => + resolveRequestedLoginProviderOrThrow(providers, "google-antigravity"), + ).toThrowError( + 'Unknown provider "google-antigravity". Loaded providers: google-gemini-cli, qwen-portal. Verify plugins via `openclaw plugins list --json`.', + ); + }); +}); diff --git a/src/commands/models.list.e2e.test.ts b/src/commands/models.list.e2e.test.ts deleted file mode 100644 index 199ef6402de..00000000000 --- a/src/commands/models.list.e2e.test.ts +++ /dev/null @@ -1,283 +0,0 @@ -import { describe, expect, it, vi } from "vitest"; - -const loadConfig = vi.fn(); -const ensureOpenClawModelsJson = vi.fn().mockResolvedValue(undefined); -const resolveOpenClawAgentDir = vi.fn().mockReturnValue("/tmp/openclaw-agent"); -const ensureAuthProfileStore = vi.fn().mockReturnValue({ version: 1, profiles: {} }); -const listProfilesForProvider = vi.fn().mockReturnValue([]); -const resolveAuthProfileDisplayLabel = vi.fn(({ profileId }: { profileId: string }) => profileId); -const resolveAuthStorePathForDisplay = vi - .fn() - .mockReturnValue("/tmp/openclaw-agent/auth-profiles.json"); -const resolveProfileUnusableUntilForDisplay = vi.fn().mockReturnValue(null); -const resolveEnvApiKey = vi.fn().mockReturnValue(undefined); -const resolveAwsSdkEnvVarName = vi.fn().mockReturnValue(undefined); -const getCustomProviderApiKey = vi.fn().mockReturnValue(undefined); -const modelRegistryState = { - models: [] as Array>, - available: [] as Array>, -}; - -vi.mock("../config/config.js", () => ({ - CONFIG_PATH: "/tmp/openclaw.json", - STATE_DIR: "/tmp/openclaw-state", - loadConfig, -})); - -vi.mock("../agents/models-config.js", () => ({ - ensureOpenClawModelsJson, -})); - -vi.mock("../agents/agent-paths.js", () => ({ - resolveOpenClawAgentDir, -})); - -vi.mock("../agents/auth-profiles.js", () => ({ - ensureAuthProfileStore, - listProfilesForProvider, - resolveAuthProfileDisplayLabel, - resolveAuthStorePathForDisplay, - resolveProfileUnusableUntilForDisplay, -})); - -vi.mock("../agents/model-auth.js", () => ({ - resolveEnvApiKey, - resolveAwsSdkEnvVarName, - getCustomProviderApiKey, -})); - -vi.mock("@mariozechner/pi-coding-agent", () => ({ - AuthStorage: class {}, - ModelRegistry: class { - getAll() { - return modelRegistryState.models; - } - getAvailable() { - return modelRegistryState.available; - } - }, -})); - -function makeRuntime() { - return { - log: vi.fn(), - error: vi.fn(), - }; -} - -describe("models list/status", () => { - it("models status resolves z.ai alias to canonical zai", async () => { - loadConfig.mockReturnValue({ - agents: { defaults: { model: "z.ai/glm-4.7" } }, - }); - const runtime = makeRuntime(); - - const { modelsStatusCommand } = await import("./models/list.js"); - await modelsStatusCommand({ json: true }, runtime); - - expect(runtime.log).toHaveBeenCalledTimes(1); - const payload = JSON.parse(String(runtime.log.mock.calls[0]?.[0])); - expect(payload.resolvedDefault).toBe("zai/glm-4.7"); - }); - - it("models status plain outputs canonical zai model", async () => { - loadConfig.mockReturnValue({ - agents: { defaults: { model: "z.ai/glm-4.7" } }, - }); - const runtime = makeRuntime(); - - const { modelsStatusCommand } = await import("./models/list.js"); - await modelsStatusCommand({ plain: true }, runtime); - - expect(runtime.log).toHaveBeenCalledTimes(1); - expect(runtime.log.mock.calls[0]?.[0]).toBe("zai/glm-4.7"); - }); - - it("models list outputs canonical zai key for configured z.ai model", async () => { - loadConfig.mockReturnValue({ - agents: { defaults: { model: "z.ai/glm-4.7" } }, - }); - const runtime = makeRuntime(); - - const model = { - provider: "zai", - id: "glm-4.7", - name: "GLM-4.7", - input: ["text"], - baseUrl: "https://api.z.ai/v1", - contextWindow: 128000, - }; - - modelRegistryState.models = [model]; - modelRegistryState.available = [model]; - - const { modelsListCommand } = await import("./models/list.js"); - await modelsListCommand({ json: true }, runtime); - - expect(runtime.log).toHaveBeenCalledTimes(1); - const payload = JSON.parse(String(runtime.log.mock.calls[0]?.[0])); - expect(payload.models[0]?.key).toBe("zai/glm-4.7"); - }); - - it("models list plain outputs canonical zai key", async () => { - loadConfig.mockReturnValue({ - agents: { defaults: { model: "z.ai/glm-4.7" } }, - }); - const runtime = makeRuntime(); - - const model = { - provider: "zai", - id: "glm-4.7", - name: "GLM-4.7", - input: ["text"], - baseUrl: "https://api.z.ai/v1", - contextWindow: 128000, - }; - - modelRegistryState.models = [model]; - modelRegistryState.available = [model]; - - const { modelsListCommand } = await import("./models/list.js"); - await modelsListCommand({ plain: true }, runtime); - - expect(runtime.log).toHaveBeenCalledTimes(1); - expect(runtime.log.mock.calls[0]?.[0]).toBe("zai/glm-4.7"); - }); - - it("models list provider filter normalizes z.ai alias", async () => { - loadConfig.mockReturnValue({ - agents: { defaults: { model: "z.ai/glm-4.7" } }, - }); - const runtime = makeRuntime(); - - const models = [ - { - provider: "zai", - id: "glm-4.7", - name: "GLM-4.7", - input: ["text"], - baseUrl: "https://api.z.ai/v1", - contextWindow: 128000, - }, - { - provider: "openai", - id: "gpt-4.1-mini", - name: "GPT-4.1 mini", - input: ["text"], - baseUrl: "https://api.openai.com/v1", - contextWindow: 128000, - }, - ]; - - modelRegistryState.models = models; - modelRegistryState.available = models; - - const { modelsListCommand } = await import("./models/list.js"); - await modelsListCommand({ all: true, provider: "z.ai", json: true }, runtime); - - expect(runtime.log).toHaveBeenCalledTimes(1); - const payload = JSON.parse(String(runtime.log.mock.calls[0]?.[0])); - expect(payload.count).toBe(1); - expect(payload.models[0]?.key).toBe("zai/glm-4.7"); - }); - - it("models list provider filter normalizes Z.AI alias casing", async () => { - loadConfig.mockReturnValue({ - agents: { defaults: { model: "z.ai/glm-4.7" } }, - }); - const runtime = makeRuntime(); - - const models = [ - { - provider: "zai", - id: "glm-4.7", - name: "GLM-4.7", - input: ["text"], - baseUrl: "https://api.z.ai/v1", - contextWindow: 128000, - }, - { - provider: "openai", - id: "gpt-4.1-mini", - name: "GPT-4.1 mini", - input: ["text"], - baseUrl: "https://api.openai.com/v1", - contextWindow: 128000, - }, - ]; - - modelRegistryState.models = models; - modelRegistryState.available = models; - - const { modelsListCommand } = await import("./models/list.js"); - await modelsListCommand({ all: true, provider: "Z.AI", json: true }, runtime); - - expect(runtime.log).toHaveBeenCalledTimes(1); - const payload = JSON.parse(String(runtime.log.mock.calls[0]?.[0])); - expect(payload.count).toBe(1); - expect(payload.models[0]?.key).toBe("zai/glm-4.7"); - }); - - it("models list provider filter normalizes z-ai alias", async () => { - loadConfig.mockReturnValue({ - agents: { defaults: { model: "z.ai/glm-4.7" } }, - }); - const runtime = makeRuntime(); - - const models = [ - { - provider: "zai", - id: "glm-4.7", - name: "GLM-4.7", - input: ["text"], - baseUrl: "https://api.z.ai/v1", - contextWindow: 128000, - }, - { - provider: "openai", - id: "gpt-4.1-mini", - name: "GPT-4.1 mini", - input: ["text"], - baseUrl: "https://api.openai.com/v1", - contextWindow: 128000, - }, - ]; - - modelRegistryState.models = models; - modelRegistryState.available = models; - - const { modelsListCommand } = await import("./models/list.js"); - await modelsListCommand({ all: true, provider: "z-ai", json: true }, runtime); - - expect(runtime.log).toHaveBeenCalledTimes(1); - const payload = JSON.parse(String(runtime.log.mock.calls[0]?.[0])); - expect(payload.count).toBe(1); - expect(payload.models[0]?.key).toBe("zai/glm-4.7"); - }); - - it("models list marks auth as unavailable when ZAI key is missing", async () => { - loadConfig.mockReturnValue({ - agents: { defaults: { model: "z.ai/glm-4.7" } }, - }); - const runtime = makeRuntime(); - - const model = { - provider: "zai", - id: "glm-4.7", - name: "GLM-4.7", - input: ["text"], - baseUrl: "https://api.z.ai/v1", - contextWindow: 128000, - }; - - modelRegistryState.models = [model]; - modelRegistryState.available = []; - - const { modelsListCommand } = await import("./models/list.js"); - await modelsListCommand({ all: true, json: true }, runtime); - - expect(runtime.log).toHaveBeenCalledTimes(1); - const payload = JSON.parse(String(runtime.log.mock.calls[0]?.[0])); - expect(payload.models[0]?.available).toBe(false); - }); -}); diff --git a/src/commands/models.list.test.ts b/src/commands/models.list.test.ts new file mode 100644 index 00000000000..556778b981e --- /dev/null +++ b/src/commands/models.list.test.ts @@ -0,0 +1,750 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; + +const loadConfig = vi.fn(); +const ensureOpenClawModelsJson = vi.fn().mockResolvedValue(undefined); +const resolveOpenClawAgentDir = vi.fn().mockReturnValue("/tmp/openclaw-agent"); +const ensureAuthProfileStore = vi.fn().mockReturnValue({ version: 1, profiles: {} }); +const listProfilesForProvider = vi.fn().mockReturnValue([]); +const resolveAuthProfileDisplayLabel = vi.fn(({ profileId }: { profileId: string }) => profileId); +const resolveAuthStorePathForDisplay = vi + .fn() + .mockReturnValue("/tmp/openclaw-agent/auth-profiles.json"); +const resolveProfileUnusableUntilForDisplay = vi.fn().mockReturnValue(null); +const resolveEnvApiKey = vi.fn().mockReturnValue(undefined); +const resolveAwsSdkEnvVarName = vi.fn().mockReturnValue(undefined); +const getCustomProviderApiKey = vi.fn().mockReturnValue(undefined); +const modelRegistryState = { + models: [] as Array>, + available: [] as Array>, + getAllError: undefined as unknown, + getAvailableError: undefined as unknown, +}; +let previousExitCode: number | undefined; + +vi.mock("../config/config.js", () => ({ + CONFIG_PATH: "/tmp/openclaw.json", + STATE_DIR: "/tmp/openclaw-state", + loadConfig, +})); + +vi.mock("../agents/models-config.js", () => ({ + ensureOpenClawModelsJson, +})); + +vi.mock("../agents/agent-paths.js", () => ({ + resolveOpenClawAgentDir, +})); + +vi.mock("../agents/auth-profiles.js", () => ({ + ensureAuthProfileStore, + listProfilesForProvider, + resolveAuthProfileDisplayLabel, + resolveAuthStorePathForDisplay, + resolveProfileUnusableUntilForDisplay, +})); + +vi.mock("../agents/model-auth.js", () => ({ + resolveEnvApiKey, + resolveAwsSdkEnvVarName, + getCustomProviderApiKey, +})); + +vi.mock("@mariozechner/pi-coding-agent", async () => { + class MockAuthStorage {} + + class MockModelRegistry { + find(provider: string, id: string) { + const found = + modelRegistryState.models.find((model) => model.provider === provider && model.id === id) ?? + null; + return found; + } + + getAll() { + if (modelRegistryState.getAllError !== undefined) { + throw modelRegistryState.getAllError; + } + return modelRegistryState.models; + } + + getAvailable() { + if (modelRegistryState.getAvailableError !== undefined) { + throw modelRegistryState.getAvailableError; + } + return modelRegistryState.available; + } + } + + return { + AuthStorage: MockAuthStorage, + ModelRegistry: MockModelRegistry, + }; +}); + +function makeRuntime() { + return { + log: vi.fn(), + error: vi.fn(), + }; +} + +beforeEach(() => { + previousExitCode = process.exitCode; + process.exitCode = undefined; + modelRegistryState.getAllError = undefined; + modelRegistryState.getAvailableError = undefined; + listProfilesForProvider.mockReturnValue([]); +}); + +afterEach(() => { + process.exitCode = previousExitCode; +}); + +describe("models list/status", () => { + it("models list outputs canonical zai key for configured z.ai model", async () => { + loadConfig.mockReturnValue({ + agents: { defaults: { model: "z.ai/glm-4.7" } }, + }); + const runtime = makeRuntime(); + + const model = { + provider: "zai", + id: "glm-4.7", + name: "GLM-4.7", + input: ["text"], + baseUrl: "https://api.z.ai/v1", + contextWindow: 128000, + }; + + modelRegistryState.models = [model]; + modelRegistryState.available = [model]; + + const { modelsListCommand } = await import("./models/list.list-command.js"); + await modelsListCommand({ json: true }, runtime); + + expect(runtime.log).toHaveBeenCalledTimes(1); + const payload = JSON.parse(String(runtime.log.mock.calls[0]?.[0])); + expect(payload.models[0]?.key).toBe("zai/glm-4.7"); + }); + + it("models list plain outputs canonical zai key", async () => { + loadConfig.mockReturnValue({ + agents: { defaults: { model: "z.ai/glm-4.7" } }, + }); + const runtime = makeRuntime(); + + const model = { + provider: "zai", + id: "glm-4.7", + name: "GLM-4.7", + input: ["text"], + baseUrl: "https://api.z.ai/v1", + contextWindow: 128000, + }; + + modelRegistryState.models = [model]; + modelRegistryState.available = [model]; + + const { modelsListCommand } = await import("./models/list.list-command.js"); + await modelsListCommand({ plain: true }, runtime); + + expect(runtime.log).toHaveBeenCalledTimes(1); + expect(runtime.log.mock.calls[0]?.[0]).toBe("zai/glm-4.7"); + }); + + it("models list provider filter normalizes z.ai alias", async () => { + loadConfig.mockReturnValue({ + agents: { defaults: { model: "z.ai/glm-4.7" } }, + }); + const runtime = makeRuntime(); + + const models = [ + { + provider: "zai", + id: "glm-4.7", + name: "GLM-4.7", + input: ["text"], + baseUrl: "https://api.z.ai/v1", + contextWindow: 128000, + }, + { + provider: "openai", + id: "gpt-4.1-mini", + name: "GPT-4.1 mini", + input: ["text"], + baseUrl: "https://api.openai.com/v1", + contextWindow: 128000, + }, + ]; + + modelRegistryState.models = models; + modelRegistryState.available = models; + + const { modelsListCommand } = await import("./models/list.list-command.js"); + await modelsListCommand({ all: true, provider: "z.ai", json: true }, runtime); + + expect(runtime.log).toHaveBeenCalledTimes(1); + const payload = JSON.parse(String(runtime.log.mock.calls[0]?.[0])); + expect(payload.count).toBe(1); + expect(payload.models[0]?.key).toBe("zai/glm-4.7"); + }); + + it("models list provider filter normalizes Z.AI alias casing", async () => { + loadConfig.mockReturnValue({ + agents: { defaults: { model: "z.ai/glm-4.7" } }, + }); + const runtime = makeRuntime(); + + const models = [ + { + provider: "zai", + id: "glm-4.7", + name: "GLM-4.7", + input: ["text"], + baseUrl: "https://api.z.ai/v1", + contextWindow: 128000, + }, + { + provider: "openai", + id: "gpt-4.1-mini", + name: "GPT-4.1 mini", + input: ["text"], + baseUrl: "https://api.openai.com/v1", + contextWindow: 128000, + }, + ]; + + modelRegistryState.models = models; + modelRegistryState.available = models; + + const { modelsListCommand } = await import("./models/list.list-command.js"); + await modelsListCommand({ all: true, provider: "Z.AI", json: true }, runtime); + + expect(runtime.log).toHaveBeenCalledTimes(1); + const payload = JSON.parse(String(runtime.log.mock.calls[0]?.[0])); + expect(payload.count).toBe(1); + expect(payload.models[0]?.key).toBe("zai/glm-4.7"); + }); + + it("models list provider filter normalizes z-ai alias", async () => { + loadConfig.mockReturnValue({ + agents: { defaults: { model: "z.ai/glm-4.7" } }, + }); + const runtime = makeRuntime(); + + const models = [ + { + provider: "zai", + id: "glm-4.7", + name: "GLM-4.7", + input: ["text"], + baseUrl: "https://api.z.ai/v1", + contextWindow: 128000, + }, + { + provider: "openai", + id: "gpt-4.1-mini", + name: "GPT-4.1 mini", + input: ["text"], + baseUrl: "https://api.openai.com/v1", + contextWindow: 128000, + }, + ]; + + modelRegistryState.models = models; + modelRegistryState.available = models; + + const { modelsListCommand } = await import("./models/list.list-command.js"); + await modelsListCommand({ all: true, provider: "z-ai", json: true }, runtime); + + expect(runtime.log).toHaveBeenCalledTimes(1); + const payload = JSON.parse(String(runtime.log.mock.calls[0]?.[0])); + expect(payload.count).toBe(1); + expect(payload.models[0]?.key).toBe("zai/glm-4.7"); + }); + + it("models list marks auth as unavailable when ZAI key is missing", async () => { + loadConfig.mockReturnValue({ + agents: { defaults: { model: "z.ai/glm-4.7" } }, + }); + const runtime = makeRuntime(); + + const model = { + provider: "zai", + id: "glm-4.7", + name: "GLM-4.7", + input: ["text"], + baseUrl: "https://api.z.ai/v1", + contextWindow: 128000, + }; + + modelRegistryState.models = [model]; + modelRegistryState.available = []; + + const { modelsListCommand } = await import("./models/list.list-command.js"); + await modelsListCommand({ all: true, json: true }, runtime); + + expect(runtime.log).toHaveBeenCalledTimes(1); + const payload = JSON.parse(String(runtime.log.mock.calls[0]?.[0])); + expect(payload.models[0]?.available).toBe(false); + }); + + it("models list resolves antigravity opus 4.6 thinking from 4.5 template", async () => { + loadConfig.mockReturnValue({ + agents: { + defaults: { + model: "google-antigravity/claude-opus-4-6-thinking", + models: { + "google-antigravity/claude-opus-4-6-thinking": {}, + }, + }, + }, + }); + const runtime = makeRuntime(); + + modelRegistryState.models = [ + { + provider: "google-antigravity", + id: "claude-opus-4-5-thinking", + name: "Claude Opus 4.5 Thinking", + api: "google-gemini-cli", + input: ["text", "image"], + baseUrl: "https://daily-cloudcode-pa.sandbox.googleapis.com", + contextWindow: 200000, + maxTokens: 64000, + reasoning: true, + cost: { input: 5, output: 25, cacheRead: 0.5, cacheWrite: 6.25 }, + }, + ]; + modelRegistryState.available = []; + + const { modelsListCommand } = await import("./models/list.list-command.js"); + await modelsListCommand({ json: true }, runtime); + + expect(runtime.log).toHaveBeenCalledTimes(1); + const payload = JSON.parse(String(runtime.log.mock.calls[0]?.[0])); + expect(payload.models[0]?.key).toBe("google-antigravity/claude-opus-4-6-thinking"); + expect(payload.models[0]?.missing).toBe(false); + expect(payload.models[0]?.tags).toContain("default"); + expect(payload.models[0]?.tags).toContain("configured"); + }); + + it("models list resolves antigravity opus 4.6 (non-thinking) from 4.5 template", async () => { + loadConfig.mockReturnValue({ + agents: { + defaults: { + model: "google-antigravity/claude-opus-4-6", + models: { + "google-antigravity/claude-opus-4-6": {}, + }, + }, + }, + }); + const runtime = makeRuntime(); + + modelRegistryState.models = [ + { + provider: "google-antigravity", + id: "claude-opus-4-5", + name: "Claude Opus 4.5", + api: "google-gemini-cli", + input: ["text", "image"], + baseUrl: "https://daily-cloudcode-pa.sandbox.googleapis.com", + contextWindow: 200000, + maxTokens: 64000, + reasoning: true, + cost: { input: 5, output: 25, cacheRead: 0.5, cacheWrite: 6.25 }, + }, + ]; + modelRegistryState.available = []; + + const { modelsListCommand } = await import("./models/list.list-command.js"); + await modelsListCommand({ json: true }, runtime); + + expect(runtime.log).toHaveBeenCalledTimes(1); + const payload = JSON.parse(String(runtime.log.mock.calls[0]?.[0])); + expect(payload.models[0]?.key).toBe("google-antigravity/claude-opus-4-6"); + expect(payload.models[0]?.missing).toBe(false); + expect(payload.models[0]?.tags).toContain("default"); + expect(payload.models[0]?.tags).toContain("configured"); + }); + + it("models list marks synthesized antigravity opus 4.6 thinking as available when template is available", async () => { + loadConfig.mockReturnValue({ + agents: { + defaults: { + model: "google-antigravity/claude-opus-4-6-thinking", + models: { + "google-antigravity/claude-opus-4-6-thinking": {}, + }, + }, + }, + }); + const runtime = makeRuntime(); + + const template = { + provider: "google-antigravity", + id: "claude-opus-4-5-thinking", + name: "Claude Opus 4.5 Thinking", + api: "google-gemini-cli", + input: ["text", "image"], + baseUrl: "https://daily-cloudcode-pa.sandbox.googleapis.com", + contextWindow: 200000, + maxTokens: 64000, + reasoning: true, + cost: { input: 5, output: 25, cacheRead: 0.5, cacheWrite: 6.25 }, + }; + modelRegistryState.models = [template]; + modelRegistryState.available = [template]; + + const { modelsListCommand } = await import("./models/list.list-command.js"); + await modelsListCommand({ json: true }, runtime); + + expect(runtime.log).toHaveBeenCalledTimes(1); + const payload = JSON.parse(String(runtime.log.mock.calls[0]?.[0])); + expect(payload.models[0]?.key).toBe("google-antigravity/claude-opus-4-6-thinking"); + expect(payload.models[0]?.missing).toBe(false); + expect(payload.models[0]?.available).toBe(true); + }); + + it("models list marks synthesized antigravity opus 4.6 (non-thinking) as available when template is available", async () => { + loadConfig.mockReturnValue({ + agents: { + defaults: { + model: "google-antigravity/claude-opus-4-6", + models: { + "google-antigravity/claude-opus-4-6": {}, + }, + }, + }, + }); + const runtime = makeRuntime(); + + const template = { + provider: "google-antigravity", + id: "claude-opus-4-5", + name: "Claude Opus 4.5", + api: "google-gemini-cli", + input: ["text", "image"], + baseUrl: "https://daily-cloudcode-pa.sandbox.googleapis.com", + contextWindow: 200000, + maxTokens: 64000, + reasoning: true, + cost: { input: 5, output: 25, cacheRead: 0.5, cacheWrite: 6.25 }, + }; + modelRegistryState.models = [template]; + modelRegistryState.available = [template]; + + const { modelsListCommand } = await import("./models/list.list-command.js"); + await modelsListCommand({ json: true }, runtime); + + expect(runtime.log).toHaveBeenCalledTimes(1); + const payload = JSON.parse(String(runtime.log.mock.calls[0]?.[0])); + expect(payload.models[0]?.key).toBe("google-antigravity/claude-opus-4-6"); + expect(payload.models[0]?.missing).toBe(false); + expect(payload.models[0]?.available).toBe(true); + }); + + it("models list prefers registry availability over provider auth heuristics", async () => { + loadConfig.mockReturnValue({ + agents: { + defaults: { + model: "google-antigravity/claude-opus-4-6-thinking", + models: { + "google-antigravity/claude-opus-4-6-thinking": {}, + }, + }, + }, + }); + listProfilesForProvider.mockImplementation((_: unknown, provider: string) => + provider === "google-antigravity" + ? ([{ id: "profile-1" }] as Array>) + : [], + ); + const runtime = makeRuntime(); + + const template = { + provider: "google-antigravity", + id: "claude-opus-4-5-thinking", + name: "Claude Opus 4.5 Thinking", + api: "google-gemini-cli", + input: ["text", "image"], + baseUrl: "https://daily-cloudcode-pa.sandbox.googleapis.com", + contextWindow: 200000, + maxTokens: 64000, + reasoning: true, + cost: { input: 5, output: 25, cacheRead: 0.5, cacheWrite: 6.25 }, + }; + modelRegistryState.models = [template]; + modelRegistryState.available = []; + + const { modelsListCommand } = await import("./models/list.list-command.js"); + await modelsListCommand({ json: true }, runtime); + + expect(runtime.log).toHaveBeenCalledTimes(1); + const payload = JSON.parse(String(runtime.log.mock.calls[0]?.[0])); + expect(payload.models[0]?.key).toBe("google-antigravity/claude-opus-4-6-thinking"); + expect(payload.models[0]?.missing).toBe(false); + expect(payload.models[0]?.available).toBe(false); + listProfilesForProvider.mockReturnValue([]); + }); + + it("models list falls back to auth heuristics when registry availability is unavailable", async () => { + loadConfig.mockReturnValue({ + agents: { + defaults: { + model: "google-antigravity/claude-opus-4-6-thinking", + models: { + "google-antigravity/claude-opus-4-6-thinking": {}, + }, + }, + }, + }); + listProfilesForProvider.mockImplementation((_: unknown, provider: string) => + provider === "google-antigravity" + ? ([{ id: "profile-1" }] as Array>) + : [], + ); + modelRegistryState.getAvailableError = Object.assign( + new Error("availability unsupported: getAvailable failed"), + { code: "MODEL_AVAILABILITY_UNAVAILABLE" }, + ); + const runtime = makeRuntime(); + + modelRegistryState.models = [ + { + provider: "google-antigravity", + id: "claude-opus-4-5-thinking", + name: "Claude Opus 4.5 Thinking", + api: "google-gemini-cli", + input: ["text", "image"], + baseUrl: "https://daily-cloudcode-pa.sandbox.googleapis.com", + contextWindow: 200000, + maxTokens: 64000, + reasoning: true, + cost: { input: 5, output: 25, cacheRead: 0.5, cacheWrite: 6.25 }, + }, + ]; + modelRegistryState.available = []; + + const { modelsListCommand } = await import("./models/list.list-command.js"); + await modelsListCommand({ json: true }, runtime); + + expect(runtime.error).toHaveBeenCalledTimes(1); + expect(runtime.error.mock.calls[0]?.[0]).toContain("falling back to auth heuristics"); + expect(runtime.error.mock.calls[0]?.[0]).toContain("getAvailable failed"); + expect(runtime.log).toHaveBeenCalledTimes(1); + const payload = JSON.parse(String(runtime.log.mock.calls[0]?.[0])); + expect(payload.models[0]?.key).toBe("google-antigravity/claude-opus-4-6-thinking"); + expect(payload.models[0]?.missing).toBe(false); + expect(payload.models[0]?.available).toBe(true); + }); + + it("models list falls back to auth heuristics when getAvailable returns invalid shape", async () => { + loadConfig.mockReturnValue({ + agents: { + defaults: { + model: "google-antigravity/claude-opus-4-6-thinking", + models: { + "google-antigravity/claude-opus-4-6-thinking": {}, + }, + }, + }, + }); + listProfilesForProvider.mockImplementation((_: unknown, provider: string) => + provider === "google-antigravity" + ? ([{ id: "profile-1" }] as Array>) + : [], + ); + modelRegistryState.available = { bad: true } as unknown as Array>; + const runtime = makeRuntime(); + + modelRegistryState.models = [ + { + provider: "google-antigravity", + id: "claude-opus-4-5-thinking", + name: "Claude Opus 4.5 Thinking", + api: "google-gemini-cli", + input: ["text", "image"], + baseUrl: "https://daily-cloudcode-pa.sandbox.googleapis.com", + contextWindow: 200000, + maxTokens: 64000, + reasoning: true, + cost: { input: 5, output: 25, cacheRead: 0.5, cacheWrite: 6.25 }, + }, + ]; + + const { modelsListCommand } = await import("./models/list.list-command.js"); + await modelsListCommand({ json: true }, runtime); + + expect(runtime.error).toHaveBeenCalledTimes(1); + expect(runtime.error.mock.calls[0]?.[0]).toContain("falling back to auth heuristics"); + expect(runtime.error.mock.calls[0]?.[0]).toContain("non-array value"); + expect(runtime.log).toHaveBeenCalledTimes(1); + const payload = JSON.parse(String(runtime.log.mock.calls[0]?.[0])); + expect(payload.models[0]?.key).toBe("google-antigravity/claude-opus-4-6-thinking"); + expect(payload.models[0]?.missing).toBe(false); + expect(payload.models[0]?.available).toBe(true); + }); + + it("models list falls back to auth heuristics when getAvailable throws", async () => { + loadConfig.mockReturnValue({ + agents: { + defaults: { + model: "google-antigravity/claude-opus-4-6-thinking", + models: { + "google-antigravity/claude-opus-4-6-thinking": {}, + }, + }, + }, + }); + listProfilesForProvider.mockImplementation((_: unknown, provider: string) => + provider === "google-antigravity" + ? ([{ id: "profile-1" }] as Array>) + : [], + ); + modelRegistryState.getAvailableError = new Error( + "availability unsupported: getAvailable failed", + ); + const runtime = makeRuntime(); + + modelRegistryState.models = [ + { + provider: "google-antigravity", + id: "claude-opus-4-5-thinking", + name: "Claude Opus 4.5 Thinking", + api: "google-gemini-cli", + input: ["text", "image"], + baseUrl: "https://daily-cloudcode-pa.sandbox.googleapis.com", + contextWindow: 200000, + maxTokens: 64000, + reasoning: true, + cost: { input: 5, output: 25, cacheRead: 0.5, cacheWrite: 6.25 }, + }, + ]; + modelRegistryState.available = []; + + const { modelsListCommand } = await import("./models/list.list-command.js"); + await modelsListCommand({ json: true }, runtime); + + expect(runtime.error).toHaveBeenCalledTimes(1); + expect(runtime.error.mock.calls[0]?.[0]).toContain("falling back to auth heuristics"); + expect(runtime.error.mock.calls[0]?.[0]).toContain( + "availability unsupported: getAvailable failed", + ); + expect(runtime.log).toHaveBeenCalledTimes(1); + const payload = JSON.parse(String(runtime.log.mock.calls[0]?.[0])); + expect(payload.models[0]?.key).toBe("google-antigravity/claude-opus-4-6-thinking"); + expect(payload.models[0]?.missing).toBe(false); + expect(payload.models[0]?.available).toBe(true); + }); + + it("models list does not treat availability-unavailable code as discovery fallback", async () => { + loadConfig.mockReturnValue({ + agents: { + defaults: { + model: "google-antigravity/claude-opus-4-6-thinking", + models: { + "google-antigravity/claude-opus-4-6-thinking": {}, + }, + }, + }, + }); + modelRegistryState.getAllError = Object.assign(new Error("model discovery failed"), { + code: "MODEL_AVAILABILITY_UNAVAILABLE", + }); + const runtime = makeRuntime(); + + const { modelsListCommand } = await import("./models/list.list-command.js"); + await modelsListCommand({ json: true }, runtime); + + expect(runtime.error).toHaveBeenCalledTimes(1); + expect(runtime.error.mock.calls[0]?.[0]).toContain("Model registry unavailable:"); + expect(runtime.error.mock.calls[0]?.[0]).toContain("model discovery failed"); + expect(runtime.error.mock.calls[0]?.[0]).not.toContain("configured models may appear missing"); + expect(runtime.log).not.toHaveBeenCalled(); + expect(process.exitCode).toBe(1); + }); + + it("models list fails fast when registry model discovery is unavailable", async () => { + loadConfig.mockReturnValue({ + agents: { + defaults: { + model: "google-antigravity/claude-opus-4-6-thinking", + models: { + "google-antigravity/claude-opus-4-6-thinking": {}, + }, + }, + }, + }); + listProfilesForProvider.mockImplementation((_: unknown, provider: string) => + provider === "google-antigravity" + ? ([{ id: "profile-1" }] as Array>) + : [], + ); + modelRegistryState.getAllError = Object.assign(new Error("model discovery unavailable"), { + code: "MODEL_DISCOVERY_UNAVAILABLE", + }); + const runtime = makeRuntime(); + + modelRegistryState.models = []; + modelRegistryState.available = []; + + const { modelsListCommand } = await import("./models/list.list-command.js"); + await modelsListCommand({ json: true }, runtime); + + expect(runtime.error).toHaveBeenCalledTimes(1); + expect(runtime.error.mock.calls[0]?.[0]).toContain("Model registry unavailable:"); + expect(runtime.error.mock.calls[0]?.[0]).toContain("model discovery unavailable"); + expect(runtime.log).not.toHaveBeenCalled(); + expect(process.exitCode).toBe(1); + }); + + it("loadModelRegistry throws when model discovery is unavailable", async () => { + modelRegistryState.getAllError = Object.assign(new Error("model discovery unavailable"), { + code: "MODEL_DISCOVERY_UNAVAILABLE", + }); + modelRegistryState.available = [ + { + provider: "google-antigravity", + id: "claude-opus-4-5-thinking", + name: "Claude Opus 4.5 Thinking", + api: "google-gemini-cli", + input: ["text", "image"], + baseUrl: "https://daily-cloudcode-pa.sandbox.googleapis.com", + contextWindow: 200000, + maxTokens: 64000, + reasoning: true, + cost: { input: 5, output: 25, cacheRead: 0.5, cacheWrite: 6.25 }, + }, + ]; + + const { loadModelRegistry } = await import("./models/list.registry.js"); + await expect(loadModelRegistry({})).rejects.toThrow("model discovery unavailable"); + }); + + it("toModelRow does not crash without cfg/authStore when availability is undefined", async () => { + const { toModelRow } = await import("./models/list.registry.js"); + + const row = toModelRow({ + model: { + provider: "google-antigravity", + id: "claude-opus-4-6-thinking", + name: "Claude Opus 4.6 Thinking", + api: "google-gemini-cli", + input: ["text", "image"], + baseUrl: "https://daily-cloudcode-pa.sandbox.googleapis.com", + contextWindow: 200000, + maxTokens: 64000, + reasoning: true, + cost: { input: 5, output: 25, cacheRead: 0.5, cacheWrite: 6.25 }, + }, + key: "google-antigravity/claude-opus-4-6-thinking", + tags: [], + availableKeys: undefined, + }); + + expect(row.missing).toBe(false); + expect(row.available).toBe(false); + }); +}); diff --git a/src/commands/models/auth.ts b/src/commands/models/auth.ts index 146c1d2693f..69dd84dde05 100644 --- a/src/commands/models/auth.ts +++ b/src/commands/models/auth.ts @@ -1,10 +1,6 @@ import { confirm as clackConfirm, select as clackSelect, text as clackText } from "@clack/prompts"; import type { AuthProfileCredential } from "../../agents/auth-profiles/types.js"; -import type { - ProviderAuthMethod, - ProviderAuthResult, - ProviderPlugin, -} from "../../plugins/types.js"; +import type { ProviderAuthResult, ProviderPlugin } from "../../plugins/types.js"; import type { RuntimeEnv } from "../../runtime.js"; import { resolveAgentDir, @@ -16,7 +12,7 @@ import { normalizeProviderId } from "../../agents/model-selection.js"; import { resolveDefaultAgentWorkspaceDir } from "../../agents/workspace.js"; import { formatCliCommand } from "../../cli/command-format.js"; import { parseDurationMs } from "../../cli/parse-duration.js"; -import { readConfigFileSnapshot, type OpenClawConfig } from "../../config/config.js"; +import { readConfigFileSnapshot } from "../../config/config.js"; import { logConfigUpdated } from "../../config/logging.js"; import { resolvePluginProviders } from "../../plugins/providers.js"; import { stylePromptHint, stylePromptMessage } from "../../terminal/prompt-style.js"; @@ -26,8 +22,12 @@ import { isRemoteEnvironment } from "../oauth-env.js"; import { createVpsAwareOAuthHandlers } from "../oauth-flow.js"; import { applyAuthProfileConfig } from "../onboard-auth.js"; import { openUrl } from "../onboard-helpers.js"; -import { OPENAI_CODEX_DEFAULT_MODEL } from "../openai-codex-model-default.js"; -import { loginOpenAICodexOAuth } from "../openai-codex-oauth.js"; +import { + applyDefaultModel, + mergeConfigPatch, + pickAuthMethod, + resolveProviderMatch, +} from "../provider-auth-helpers.js"; import { updateConfig } from "./shared.js"; const confirm = (params: Parameters[0]) => @@ -241,82 +241,28 @@ type LoginOptions = { setDefault?: boolean; }; -function resolveProviderMatch( +export function resolveRequestedLoginProviderOrThrow( providers: ProviderPlugin[], rawProvider?: string, ): ProviderPlugin | null { - const raw = rawProvider?.trim(); - if (!raw) { + const requested = rawProvider?.trim(); + if (!requested) { return null; } - const normalized = normalizeProviderId(raw); - return ( - providers.find((provider) => normalizeProviderId(provider.id) === normalized) ?? - providers.find( - (provider) => - provider.aliases?.some((alias) => normalizeProviderId(alias) === normalized) ?? false, - ) ?? - null + const matched = resolveProviderMatch(providers, requested); + if (matched) { + return matched; + } + const available = providers + .map((provider) => provider.id) + .filter(Boolean) + .toSorted((a, b) => a.localeCompare(b)); + const availableText = available.length > 0 ? available.join(", ") : "(none)"; + throw new Error( + `Unknown provider "${requested}". Loaded providers: ${availableText}. Verify plugins via \`${formatCliCommand("openclaw plugins list --json")}\`.`, ); } -function pickAuthMethod(provider: ProviderPlugin, rawMethod?: string): ProviderAuthMethod | null { - const raw = rawMethod?.trim(); - if (!raw) { - return null; - } - const normalized = raw.toLowerCase(); - return ( - provider.auth.find((method) => method.id.toLowerCase() === normalized) ?? - provider.auth.find((method) => method.label.toLowerCase() === normalized) ?? - null - ); -} - -function isPlainRecord(value: unknown): value is Record { - return Boolean(value && typeof value === "object" && !Array.isArray(value)); -} - -function mergeConfigPatch(base: T, patch: unknown): T { - if (!isPlainRecord(base) || !isPlainRecord(patch)) { - return patch as T; - } - - const next: Record = { ...base }; - for (const [key, value] of Object.entries(patch)) { - const existing = next[key]; - if (isPlainRecord(existing) && isPlainRecord(value)) { - next[key] = mergeConfigPatch(existing, value); - } else { - next[key] = value; - } - } - return next as T; -} - -function applyDefaultModel(cfg: OpenClawConfig, model: string): OpenClawConfig { - const models = { ...cfg.agents?.defaults?.models }; - models[model] = models[model] ?? {}; - - const existingModel = cfg.agents?.defaults?.model; - return { - ...cfg, - agents: { - ...cfg.agents, - defaults: { - ...cfg.agents?.defaults, - models, - model: { - ...(existingModel && typeof existingModel === "object" && "fallbacks" in existingModel - ? { fallbacks: (existingModel as { fallbacks?: string[] }).fallbacks } - : undefined), - primary: model, - }, - }, - }, - }; -} - function credentialMode(credential: AuthProfileCredential): "api_key" | "oauth" | "token" { if (credential.type === "api_key") { return "api_key"; @@ -344,59 +290,6 @@ export async function modelsAuthLoginCommand(opts: LoginOptions, runtime: Runtim const workspaceDir = resolveAgentWorkspaceDir(config, defaultAgentId) ?? resolveDefaultAgentWorkspaceDir(); - const prompter = createClackPrompter(); - const requestedProvider = opts.provider ? normalizeProviderId(opts.provider) : null; - if (requestedProvider === "openai-codex") { - const method = opts.method?.trim().toLowerCase(); - if (method && method !== "oauth") { - throw new Error('OpenAI Codex auth only supports --method "oauth".'); - } - - const creds = await loginOpenAICodexOAuth({ - prompter, - runtime, - isRemote: isRemoteEnvironment(), - openUrl: async (url) => { - await openUrl(url); - }, - }); - if (!creds) { - return; - } - - const profileId = "openai-codex:default"; - upsertAuthProfile({ - profileId, - credential: { - type: "oauth", - provider: "openai-codex", - ...creds, - }, - agentDir, - }); - - await updateConfig((cfg) => { - let next = applyAuthProfileConfig(cfg, { - profileId, - provider: "openai-codex", - mode: "oauth", - }); - if (opts.setDefault) { - next = applyDefaultModel(next, OPENAI_CODEX_DEFAULT_MODEL); - } - return next; - }); - - logConfigUpdated(runtime); - runtime.log(`Auth profile: ${profileId} (openai-codex/oauth)`); - runtime.log( - opts.setDefault - ? `Default model set to ${OPENAI_CODEX_DEFAULT_MODEL}` - : `Default model available: ${OPENAI_CODEX_DEFAULT_MODEL} (use --set-default to apply)`, - ); - return; - } - const providers = resolvePluginProviders({ config, workspaceDir }); if (providers.length === 0) { throw new Error( @@ -404,8 +297,10 @@ export async function modelsAuthLoginCommand(opts: LoginOptions, runtime: Runtim ); } + const prompter = createClackPrompter(); + const requestedProvider = resolveRequestedLoginProviderOrThrow(providers, opts.provider); const selectedProvider = - resolveProviderMatch(providers, opts.provider) ?? + requestedProvider ?? (await prompter .select({ message: "Select a provider", diff --git a/src/commands/models/list.errors.ts b/src/commands/models/list.errors.ts new file mode 100644 index 00000000000..3c501e095db --- /dev/null +++ b/src/commands/models/list.errors.ts @@ -0,0 +1,16 @@ +export const MODEL_AVAILABILITY_UNAVAILABLE_CODE = "MODEL_AVAILABILITY_UNAVAILABLE"; + +export function formatErrorWithStack(err: unknown): string { + if (err instanceof Error) { + return err.stack ?? `${err.name}: ${err.message}`; + } + return String(err); +} + +export function shouldFallbackToAuthHeuristics(err: unknown): boolean { + if (!(err instanceof Error)) { + return false; + } + const code = (err as { code?: unknown }).code; + return code === MODEL_AVAILABILITY_UNAVAILABLE_CODE; +} diff --git a/src/commands/models/list.list-command.forward-compat.test.ts b/src/commands/models/list.list-command.forward-compat.test.ts index 2f7f6ec2719..396509f8a31 100644 --- a/src/commands/models/list.list-command.forward-compat.test.ts +++ b/src/commands/models/list.list-command.forward-compat.test.ts @@ -4,37 +4,34 @@ const mocks = vi.hoisted(() => { const printModelTable = vi.fn(); return { loadConfig: vi.fn().mockReturnValue({ - agents: { defaults: { model: { primary: "openai-codex/gpt-5.3-codex-spark" } } }, + agents: { defaults: { model: { primary: "openai-codex/gpt-5.3-codex" } } }, models: { providers: {} }, }), ensureAuthProfileStore: vi.fn().mockReturnValue({ version: 1, profiles: {}, order: {} }), - loadModelRegistry: vi.fn().mockResolvedValue({ models: [], availableKeys: new Set() }), + loadModelRegistry: vi + .fn() + .mockResolvedValue({ models: [], availableKeys: new Set(), registry: {} }), resolveConfiguredEntries: vi.fn().mockReturnValue({ entries: [ { - key: "openai-codex/gpt-5.3-codex-spark", - ref: { provider: "openai-codex", model: "gpt-5.3-codex-spark" }, + key: "openai-codex/gpt-5.3-codex", + ref: { provider: "openai-codex", model: "gpt-5.3-codex" }, tags: new Set(["configured"]), aliases: [], }, ], }), printModelTable, - resolveModel: vi.fn().mockReturnValue({ - model: { - provider: "openai-codex", - id: "gpt-5.3-codex-spark", - name: "GPT-5.3 Codex Spark", - api: "openai-codex-responses", - baseUrl: "https://chatgpt.com/backend-api", - input: ["text"], - contextWindow: 272000, - maxTokens: 128000, - cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 }, - }, - error: undefined, - authStorage: {} as never, - modelRegistry: {} as never, + resolveForwardCompatModel: vi.fn().mockReturnValue({ + provider: "openai-codex", + id: "gpt-5.3-codex", + name: "GPT-5.3 Codex", + api: "openai-codex-responses", + baseUrl: "https://chatgpt.com/backend-api", + input: ["text"], + contextWindow: 272000, + maxTokens: 128000, + cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 }, }), }; }); @@ -68,14 +65,18 @@ vi.mock("./list.table.js", () => ({ printModelTable: mocks.printModelTable, })); -vi.mock("../../agents/pi-embedded-runner/model.js", () => ({ - resolveModel: mocks.resolveModel, -})); +vi.mock("../../agents/model-forward-compat.js", async (importOriginal) => { + const actual = await importOriginal(); + return { + ...actual, + resolveForwardCompatModel: mocks.resolveForwardCompatModel, + }; +}); import { modelsListCommand } from "./list.list-command.js"; describe("modelsListCommand forward-compat", () => { - it("does not mark configured codex spark as missing when resolveModel can build a fallback", async () => { + it("does not mark configured codex model as missing when forward-compat can build a fallback", async () => { const runtime = { log: vi.fn(), error: vi.fn() }; await modelsListCommand({ json: true }, runtime as never); @@ -87,9 +88,9 @@ describe("modelsListCommand forward-compat", () => { missing: boolean; }>; - const spark = rows.find((r) => r.key === "openai-codex/gpt-5.3-codex-spark"); - expect(spark).toBeTruthy(); - expect(spark?.missing).toBe(false); - expect(spark?.tags).not.toContain("missing"); + const codex = rows.find((r) => r.key === "openai-codex/gpt-5.3-codex"); + expect(codex).toBeTruthy(); + expect(codex?.missing).toBe(false); + expect(codex?.tags).not.toContain("missing"); }); }); diff --git a/src/commands/models/list.list-command.ts b/src/commands/models/list.list-command.ts index dcc8bf089ff..c371e85a308 100644 --- a/src/commands/models/list.list-command.ts +++ b/src/commands/models/list.list-command.ts @@ -1,14 +1,17 @@ import type { Api, Model } from "@mariozechner/pi-ai"; +import type { ModelRegistry } from "../../agents/pi-model-discovery.js"; import type { RuntimeEnv } from "../../runtime.js"; import type { ModelRow } from "./list.types.js"; import { ensureAuthProfileStore } from "../../agents/auth-profiles.js"; +import { resolveForwardCompatModel } from "../../agents/model-forward-compat.js"; import { parseModelRef } from "../../agents/model-selection.js"; import { resolveModel } from "../../agents/pi-embedded-runner/model.js"; import { loadConfig } from "../../config/config.js"; import { resolveConfiguredEntries } from "./list.configured.js"; +import { formatErrorWithStack } from "./list.errors.js"; import { loadModelRegistry, toModelRow } from "./list.registry.js"; import { printModelTable } from "./list.table.js"; -import { DEFAULT_PROVIDER, ensureFlagCompatibility, modelKey } from "./shared.js"; +import { DEFAULT_PROVIDER, ensureFlagCompatibility, isLocalBaseUrl, modelKey } from "./shared.js"; export async function modelsListCommand( opts: { @@ -33,13 +36,24 @@ export async function modelsListCommand( })(); let models: Model[] = []; + let modelRegistry: ModelRegistry | undefined; let availableKeys: Set | undefined; + let availabilityErrorMessage: string | undefined; try { const loaded = await loadModelRegistry(cfg); + modelRegistry = loaded.registry; models = loaded.models; availableKeys = loaded.availableKeys; + availabilityErrorMessage = loaded.availabilityErrorMessage; } catch (err) { - runtime.error(`Model registry unavailable: ${String(err)}`); + runtime.error(`Model registry unavailable:\n${formatErrorWithStack(err)}`); + process.exitCode = 1; + return; + } + if (availabilityErrorMessage !== undefined) { + runtime.error( + `Model availability lookup failed; falling back to auth heuristics for discovered models: ${availabilityErrorMessage}`, + ); } const modelByKey = new Map(models.map((model) => [modelKey(model.provider, model.id), model])); @@ -49,22 +63,6 @@ export async function modelsListCommand( const rows: ModelRow[] = []; - const isLocalBaseUrl = (baseUrl: string) => { - try { - const url = new URL(baseUrl); - const host = url.hostname.toLowerCase(); - return ( - host === "localhost" || - host === "127.0.0.1" || - host === "0.0.0.0" || - host === "::1" || - host.endsWith(".local") - ); - } catch { - return false; - } - }; - if (opts.all) { const sorted = [...models].toSorted((a, b) => { const p = a.provider.localeCompare(b.provider); @@ -101,12 +99,20 @@ export async function modelsListCommand( continue; } let model = modelByKey.get(entry.key); - if (!model) { - const resolved = resolveModel(entry.ref.provider, entry.ref.model, undefined, cfg); - if (resolved.model && !resolved.error) { - model = resolved.model; + if (!model && modelRegistry) { + const forwardCompat = resolveForwardCompatModel( + entry.ref.provider, + entry.ref.model, + modelRegistry, + ); + if (forwardCompat) { + model = forwardCompat; + modelByKey.set(entry.key, forwardCompat); } } + if (!model) { + model = resolveModel(entry.ref.provider, entry.ref.model, undefined, cfg).model; + } if (opts.local && model && !isLocalBaseUrl(model.baseUrl)) { continue; } diff --git a/src/commands/models/list.registry.ts b/src/commands/models/list.registry.ts index a8ff5ded52a..1edeb81980a 100644 --- a/src/commands/models/list.registry.ts +++ b/src/commands/models/list.registry.ts @@ -1,5 +1,6 @@ import type { Api, Model } from "@mariozechner/pi-ai"; import type { AuthProfileStore } from "../../agents/auth-profiles.js"; +import type { ModelRegistry } from "../../agents/pi-model-discovery.js"; import type { OpenClawConfig } from "../../config/config.js"; import type { ModelRow } from "./list.types.js"; import { resolveOpenClawAgentDir } from "../../agents/agent-paths.js"; @@ -9,28 +10,27 @@ import { resolveAwsSdkEnvVarName, resolveEnvApiKey, } from "../../agents/model-auth.js"; +import { + ANTIGRAVITY_OPUS_46_FORWARD_COMPAT_CANDIDATES, + resolveForwardCompatModel, +} from "../../agents/model-forward-compat.js"; import { ensureOpenClawModelsJson } from "../../agents/models-config.js"; -import { ensurePiAuthJsonFromAuthProfiles } from "../../agents/pi-auth-json.js"; import { discoverAuthStorage, discoverModels } from "../../agents/pi-model-discovery.js"; -import { modelKey } from "./shared.js"; +import { + formatErrorWithStack, + MODEL_AVAILABILITY_UNAVAILABLE_CODE, + shouldFallbackToAuthHeuristics, +} from "./list.errors.js"; +import { isLocalBaseUrl, modelKey } from "./shared.js"; -const isLocalBaseUrl = (baseUrl: string) => { - try { - const url = new URL(baseUrl); - const host = url.hostname.toLowerCase(); - return ( - host === "localhost" || - host === "127.0.0.1" || - host === "0.0.0.0" || - host === "::1" || - host.endsWith(".local") - ); - } catch { +const hasAuthForProvider = ( + provider: string, + cfg?: OpenClawConfig, + authStore?: AuthProfileStore, +) => { + if (!cfg || !authStore) { return false; } -}; - -const hasAuthForProvider = (provider: string, cfg: OpenClawConfig, authStore: AuthProfileStore) => { if (listProfilesForProvider(authStore, provider).length > 0) { return true; } @@ -46,16 +46,136 @@ const hasAuthForProvider = (provider: string, cfg: OpenClawConfig, authStore: Au return false; }; +function createAvailabilityUnavailableError(message: string): Error { + const err = new Error(message); + (err as { code?: string }).code = MODEL_AVAILABILITY_UNAVAILABLE_CODE; + return err; +} + +function normalizeAvailabilityError(err: unknown): Error { + if (shouldFallbackToAuthHeuristics(err) && err instanceof Error) { + return err; + } + return createAvailabilityUnavailableError( + `Model availability unavailable: getAvailable() failed.\n${formatErrorWithStack(err)}`, + ); +} + +function validateAvailableModels(availableModels: unknown): Model[] { + if (!Array.isArray(availableModels)) { + throw createAvailabilityUnavailableError( + "Model availability unavailable: getAvailable() returned a non-array value.", + ); + } + + for (const model of availableModels) { + if ( + !model || + typeof model !== "object" || + typeof (model as { provider?: unknown }).provider !== "string" || + typeof (model as { id?: unknown }).id !== "string" + ) { + throw createAvailabilityUnavailableError( + "Model availability unavailable: getAvailable() returned invalid model entries.", + ); + } + } + + return availableModels as Model[]; +} + +function loadAvailableModels(registry: ModelRegistry): Model[] { + let availableModels: unknown; + try { + availableModels = registry.getAvailable(); + } catch (err) { + throw normalizeAvailabilityError(err); + } + try { + return validateAvailableModels(availableModels); + } catch (err) { + throw normalizeAvailabilityError(err); + } +} + export async function loadModelRegistry(cfg: OpenClawConfig) { await ensureOpenClawModelsJson(cfg); const agentDir = resolveOpenClawAgentDir(); - await ensurePiAuthJsonFromAuthProfiles(agentDir); const authStorage = discoverAuthStorage(agentDir); const registry = discoverModels(authStorage, agentDir); - const models = registry.getAll(); - const availableModels = registry.getAvailable(); - const availableKeys = new Set(availableModels.map((model) => modelKey(model.provider, model.id))); - return { registry, models, availableKeys }; + const appended = appendAntigravityForwardCompatModels(registry.getAll(), registry); + const models = appended.models; + const synthesizedForwardCompat = appended.synthesizedForwardCompat; + let availableKeys: Set | undefined; + let availabilityErrorMessage: string | undefined; + + try { + const availableModels = loadAvailableModels(registry); + availableKeys = new Set(availableModels.map((model) => modelKey(model.provider, model.id))); + for (const synthesized of synthesizedForwardCompat) { + if (hasAvailableTemplate(availableKeys, synthesized.templatePrefixes)) { + availableKeys.add(synthesized.key); + } + } + } catch (err) { + if (!shouldFallbackToAuthHeuristics(err)) { + throw err; + } + + // Some providers can report model-level availability as unavailable. + // Fall back to provider-level auth heuristics when availability is undefined. + availableKeys = undefined; + if (!availabilityErrorMessage) { + availabilityErrorMessage = formatErrorWithStack(err); + } + } + return { registry, models, availableKeys, availabilityErrorMessage }; +} + +type SynthesizedForwardCompat = { + key: string; + templatePrefixes: readonly string[]; +}; + +function appendAntigravityForwardCompatModels( + models: Model[], + modelRegistry: ModelRegistry, +): { models: Model[]; synthesizedForwardCompat: SynthesizedForwardCompat[] } { + const nextModels = [...models]; + const synthesizedForwardCompat: SynthesizedForwardCompat[] = []; + + for (const candidate of ANTIGRAVITY_OPUS_46_FORWARD_COMPAT_CANDIDATES) { + const key = modelKey("google-antigravity", candidate.id); + const hasForwardCompat = nextModels.some((model) => modelKey(model.provider, model.id) === key); + if (hasForwardCompat) { + continue; + } + + const fallback = resolveForwardCompatModel("google-antigravity", candidate.id, modelRegistry); + if (!fallback) { + continue; + } + + nextModels.push(fallback); + synthesizedForwardCompat.push({ + key, + templatePrefixes: candidate.templatePrefixes, + }); + } + + return { models: nextModels, synthesizedForwardCompat }; +} + +function hasAvailableTemplate( + availableKeys: Set, + templatePrefixes: readonly string[], +): boolean { + for (const key of availableKeys) { + if (templatePrefixes.some((prefix) => key.startsWith(prefix))) { + return true; + } + } + return false; } export function toModelRow(params: { @@ -83,10 +203,14 @@ export function toModelRow(params: { const input = model.input.join("+") || "text"; const local = isLocalBaseUrl(model.baseUrl); + // Prefer model-level registry availability when present. + // Fall back to provider-level auth heuristics only if registry availability isn't available. const available = - cfg && authStore - ? hasAuthForProvider(model.provider, cfg, authStore) - : (availableKeys?.has(modelKey(model.provider, model.id)) ?? false); + availableKeys !== undefined + ? availableKeys.has(modelKey(model.provider, model.id)) + : cfg && authStore + ? hasAuthForProvider(model.provider, cfg, authStore) + : false; const aliasTags = aliases.length > 0 ? [`alias:${aliases.join(",")}`] : []; const mergedTags = new Set(tags); if (aliasTags.length > 0) { diff --git a/src/commands/models/list.status.e2e.test.ts b/src/commands/models/list.status.e2e.test.ts index daa9aa23250..2da6b3764fa 100644 --- a/src/commands/models/list.status.e2e.test.ts +++ b/src/commands/models/list.status.e2e.test.ts @@ -115,7 +115,7 @@ vi.mock("../../config/config.js", async (importOriginal) => { }; }); -import { modelsStatusCommand } from "./list.js"; +import { modelsStatusCommand } from "./list.status-command.js"; const runtime = { log: vi.fn(), diff --git a/src/commands/models/shared.ts b/src/commands/models/shared.ts index 99c64dff78f..b25be3a8926 100644 --- a/src/commands/models/shared.ts +++ b/src/commands/models/shared.ts @@ -43,6 +43,22 @@ export const formatMs = (value?: number | null) => { return `${Math.round(value / 100) / 10}s`; }; +export const isLocalBaseUrl = (baseUrl: string) => { + try { + const url = new URL(baseUrl); + const host = url.hostname.toLowerCase(); + return ( + host === "localhost" || + host === "127.0.0.1" || + host === "0.0.0.0" || + host === "::1" || + host.endsWith(".local") + ); + } catch { + return false; + } +}; + export async function updateConfig( mutator: (cfg: OpenClawConfig) => OpenClawConfig, ): Promise { diff --git a/src/commands/provider-auth-helpers.ts b/src/commands/provider-auth-helpers.ts new file mode 100644 index 00000000000..1204a3ad395 --- /dev/null +++ b/src/commands/provider-auth-helpers.ts @@ -0,0 +1,82 @@ +import type { OpenClawConfig } from "../config/config.js"; +import type { ProviderAuthMethod, ProviderPlugin } from "../plugins/types.js"; +import { normalizeProviderId } from "../agents/model-selection.js"; + +export function resolveProviderMatch( + providers: ProviderPlugin[], + rawProvider?: string, +): ProviderPlugin | null { + const raw = rawProvider?.trim(); + if (!raw) { + return null; + } + const normalized = normalizeProviderId(raw); + return ( + providers.find((provider) => normalizeProviderId(provider.id) === normalized) ?? + providers.find( + (provider) => + provider.aliases?.some((alias) => normalizeProviderId(alias) === normalized) ?? false, + ) ?? + null + ); +} + +export function pickAuthMethod( + provider: ProviderPlugin, + rawMethod?: string, +): ProviderAuthMethod | null { + const raw = rawMethod?.trim(); + if (!raw) { + return null; + } + const normalized = raw.toLowerCase(); + return ( + provider.auth.find((method) => method.id.toLowerCase() === normalized) ?? + provider.auth.find((method) => method.label.toLowerCase() === normalized) ?? + null + ); +} + +function isPlainRecord(value: unknown): value is Record { + return Boolean(value && typeof value === "object" && !Array.isArray(value)); +} + +export function mergeConfigPatch(base: T, patch: unknown): T { + if (!isPlainRecord(base) || !isPlainRecord(patch)) { + return patch as T; + } + + const next: Record = { ...base }; + for (const [key, value] of Object.entries(patch)) { + const existing = next[key]; + if (isPlainRecord(existing) && isPlainRecord(value)) { + next[key] = mergeConfigPatch(existing, value); + } else { + next[key] = value; + } + } + return next as T; +} + +export function applyDefaultModel(cfg: OpenClawConfig, model: string): OpenClawConfig { + const models = { ...cfg.agents?.defaults?.models }; + models[model] = models[model] ?? {}; + + const existingModel = cfg.agents?.defaults?.model; + return { + ...cfg, + agents: { + ...cfg.agents, + defaults: { + ...cfg.agents?.defaults, + models, + model: { + ...(existingModel && typeof existingModel === "object" && "fallbacks" in existingModel + ? { fallbacks: (existingModel as { fallbacks?: string[] }).fallbacks } + : undefined), + primary: model, + }, + }, + }, + }; +} diff --git a/src/commands/status.command.ts b/src/commands/status.command.ts index cbe5d6d78a7..04d1c505c25 100644 --- a/src/commands/status.command.ts +++ b/src/commands/status.command.ts @@ -312,6 +312,10 @@ export async function statusCommand( } if (!memory) { const slot = memoryPlugin.slot ? `plugin ${memoryPlugin.slot}` : "plugin"; + // Custom (non-built-in) memory plugins can't be probed β€” show enabled, not unavailable + if (memoryPlugin.slot && memoryPlugin.slot !== "memory-core") { + return `enabled (${slot})`; + } return muted(`enabled (${slot}) Β· unavailable`); } const parts: string[] = []; diff --git a/src/config/config.identity-defaults.test.ts b/src/config/config.identity-defaults.test.ts index fe5286fe6f7..48a6710a44a 100644 --- a/src/config/config.identity-defaults.test.ts +++ b/src/config/config.identity-defaults.test.ts @@ -1,19 +1,53 @@ import fs from "node:fs/promises"; +import os from "node:os"; import path from "node:path"; -import { afterEach, beforeEach, describe, expect, it } from "vitest"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; import { DEFAULT_AGENT_MAX_CONCURRENT, DEFAULT_SUBAGENT_MAX_CONCURRENT } from "./agent-limits.js"; import { loadConfig } from "./config.js"; -import { withTempHome } from "./test-helpers.js"; + +type HomeEnvSnapshot = { + home: string | undefined; + userProfile: string | undefined; + homeDrive: string | undefined; + homePath: string | undefined; + stateDir: string | undefined; +}; + +function snapshotHomeEnv(): HomeEnvSnapshot { + return { + home: process.env.HOME, + userProfile: process.env.USERPROFILE, + homeDrive: process.env.HOMEDRIVE, + homePath: process.env.HOMEPATH, + stateDir: process.env.OPENCLAW_STATE_DIR, + }; +} + +function restoreHomeEnv(snapshot: HomeEnvSnapshot) { + const restoreKey = (key: string, value: string | undefined) => { + if (value === undefined) { + delete process.env[key]; + } else { + process.env[key] = value; + } + }; + restoreKey("HOME", snapshot.home); + restoreKey("USERPROFILE", snapshot.userProfile); + restoreKey("HOMEDRIVE", snapshot.homeDrive); + restoreKey("HOMEPATH", snapshot.homePath); + restoreKey("OPENCLAW_STATE_DIR", snapshot.stateDir); +} describe("config identity defaults", () => { - let previousHome: string | undefined; + let fixtureRoot = ""; + let fixtureCount = 0; - beforeEach(() => { - previousHome = process.env.HOME; + beforeAll(async () => { + fixtureRoot = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-config-identity-")); }); - afterEach(() => { - process.env.HOME = previousHome; + afterAll(async () => { + await fs.rm(fixtureRoot, { recursive: true, force: true }); }); const writeAndLoadConfig = async (home: string, config: Record) => { @@ -27,6 +61,30 @@ describe("config identity defaults", () => { return loadConfig(); }; + const withTempHome = async (fn: (home: string) => Promise): Promise => { + const home = path.join(fixtureRoot, `home-${fixtureCount++}`); + await fs.mkdir(path.join(home, ".openclaw"), { recursive: true }); + + const snapshot = snapshotHomeEnv(); + process.env.HOME = home; + process.env.USERPROFILE = home; + process.env.OPENCLAW_STATE_DIR = path.join(home, ".openclaw"); + + if (process.platform === "win32") { + const match = home.match(/^([A-Za-z]:)(.*)$/); + if (match) { + process.env.HOMEDRIVE = match[1]; + process.env.HOMEPATH = match[2] || "\\"; + } + } + + try { + return await fn(home); + } finally { + restoreHomeEnv(snapshot); + } + }; + it("does not derive mention defaults and only sets ackReactionScope when identity is present", async () => { await withTempHome(async (home) => { const cfg = await writeAndLoadConfig(home, { diff --git a/src/config/config.model-compat-schema.test.ts b/src/config/config.model-compat-schema.test.ts new file mode 100644 index 00000000000..7039e44f34c --- /dev/null +++ b/src/config/config.model-compat-schema.test.ts @@ -0,0 +1,34 @@ +import { describe, expect, it } from "vitest"; +import { validateConfigObject } from "./validation.js"; + +describe("model compat config schema", () => { + it("accepts full openai-completions compat fields", () => { + const res = validateConfigObject({ + models: { + providers: { + local: { + baseUrl: "http://127.0.0.1:1234/v1", + api: "openai-completions", + models: [ + { + id: "qwen3-32b", + name: "Qwen3 32B", + compat: { + supportsUsageInStreaming: true, + supportsStrictMode: false, + thinkingFormat: "qwen", + requiresToolResultName: true, + requiresAssistantAfterToolResult: false, + requiresThinkingAsText: false, + requiresMistralToolIds: false, + }, + }, + ], + }, + }, + }, + }); + + expect(res.ok).toBe(true); + }); +}); diff --git a/src/config/config.plugin-validation.test.ts b/src/config/config.plugin-validation.test.ts index 418af2fdbac..c7389a59f27 100644 --- a/src/config/config.plugin-validation.test.ts +++ b/src/config/config.plugin-validation.test.ts @@ -1,8 +1,8 @@ import fs from "node:fs/promises"; +import os from "node:os"; import path from "node:path"; -import { describe, expect, it } from "vitest"; +import { afterAll, describe, expect, it } from "vitest"; import { validateConfigObjectWithPlugins } from "./config.js"; -import { withTempHome } from "./test-helpers.js"; async function writePluginFixture(params: { dir: string; @@ -31,145 +31,150 @@ async function writePluginFixture(params: { } describe("config plugin validation", () => { + const fixtureRoot = path.join(os.tmpdir(), "openclaw-config-plugin-validation"); + let caseIndex = 0; + + function createCaseHome() { + const home = path.join(fixtureRoot, `case-${caseIndex++}`); + return fs.mkdir(home, { recursive: true }).then(() => home); + } + const validateInHome = (home: string, raw: unknown) => { process.env.OPENCLAW_STATE_DIR = path.join(home, ".openclaw"); return validateConfigObjectWithPlugins(raw); }; + afterAll(async () => { + await fs.rm(fixtureRoot, { recursive: true, force: true }); + }); + it("rejects missing plugin load paths", async () => { - await withTempHome(async (home) => { - const missingPath = path.join(home, "missing-plugin"); - const res = validateInHome(home, { - agents: { list: [{ id: "pi" }] }, - plugins: { enabled: false, load: { paths: [missingPath] } }, - }); - expect(res.ok).toBe(false); - if (!res.ok) { - const hasIssue = res.issues.some( - (issue) => - issue.path === "plugins.load.paths" && issue.message.includes("plugin path not found"), - ); - expect(hasIssue).toBe(true); - } + const home = await createCaseHome(); + const missingPath = path.join(home, "missing-plugin"); + const res = validateInHome(home, { + agents: { list: [{ id: "pi" }] }, + plugins: { enabled: false, load: { paths: [missingPath] } }, }); + expect(res.ok).toBe(false); + if (!res.ok) { + const hasIssue = res.issues.some( + (issue) => + issue.path === "plugins.load.paths" && issue.message.includes("plugin path not found"), + ); + expect(hasIssue).toBe(true); + } }); it("rejects missing plugin ids in entries", async () => { - await withTempHome(async (home) => { - const res = validateInHome(home, { - agents: { list: [{ id: "pi" }] }, - plugins: { enabled: false, entries: { "missing-plugin": { enabled: true } } }, - }); - expect(res.ok).toBe(false); - if (!res.ok) { - expect(res.issues).toContainEqual({ - path: "plugins.entries.missing-plugin", - message: "plugin not found: missing-plugin", - }); - } + const home = await createCaseHome(); + const res = validateInHome(home, { + agents: { list: [{ id: "pi" }] }, + plugins: { enabled: false, entries: { "missing-plugin": { enabled: true } } }, }); + expect(res.ok).toBe(false); + if (!res.ok) { + expect(res.issues).toContainEqual({ + path: "plugins.entries.missing-plugin", + message: "plugin not found: missing-plugin", + }); + } }); it("rejects missing plugin ids in allow/deny/slots", async () => { - await withTempHome(async (home) => { - const res = validateInHome(home, { - agents: { list: [{ id: "pi" }] }, - plugins: { - enabled: false, - allow: ["missing-allow"], - deny: ["missing-deny"], - slots: { memory: "missing-slot" }, - }, - }); - expect(res.ok).toBe(false); - if (!res.ok) { - expect(res.issues).toEqual( - expect.arrayContaining([ - { path: "plugins.allow", message: "plugin not found: missing-allow" }, - { path: "plugins.deny", message: "plugin not found: missing-deny" }, - { path: "plugins.slots.memory", message: "plugin not found: missing-slot" }, - ]), - ); - } + const home = await createCaseHome(); + const res = validateInHome(home, { + agents: { list: [{ id: "pi" }] }, + plugins: { + enabled: false, + allow: ["missing-allow"], + deny: ["missing-deny"], + slots: { memory: "missing-slot" }, + }, }); + expect(res.ok).toBe(false); + if (!res.ok) { + expect(res.issues).toEqual( + expect.arrayContaining([ + { path: "plugins.allow", message: "plugin not found: missing-allow" }, + { path: "plugins.deny", message: "plugin not found: missing-deny" }, + { path: "plugins.slots.memory", message: "plugin not found: missing-slot" }, + ]), + ); + } }); it("surfaces plugin config diagnostics", async () => { - await withTempHome(async (home) => { - const pluginDir = path.join(home, "bad-plugin"); - await writePluginFixture({ - dir: pluginDir, - id: "bad-plugin", - schema: { - type: "object", - additionalProperties: false, - properties: { - value: { type: "boolean" }, - }, - required: ["value"], + const home = await createCaseHome(); + const pluginDir = path.join(home, "bad-plugin"); + await writePluginFixture({ + dir: pluginDir, + id: "bad-plugin", + schema: { + type: "object", + additionalProperties: false, + properties: { + value: { type: "boolean" }, }, - }); - - const res = validateInHome(home, { - agents: { list: [{ id: "pi" }] }, - plugins: { - enabled: true, - load: { paths: [pluginDir] }, - entries: { "bad-plugin": { config: { value: "nope" } } }, - }, - }); - expect(res.ok).toBe(false); - if (!res.ok) { - const hasIssue = res.issues.some( - (issue) => - issue.path === "plugins.entries.bad-plugin.config" && - issue.message.includes("invalid config"), - ); - expect(hasIssue).toBe(true); - } + required: ["value"], + }, }); + + const res = validateInHome(home, { + agents: { list: [{ id: "pi" }] }, + plugins: { + enabled: true, + load: { paths: [pluginDir] }, + entries: { "bad-plugin": { config: { value: "nope" } } }, + }, + }); + expect(res.ok).toBe(false); + if (!res.ok) { + const hasIssue = res.issues.some( + (issue) => + issue.path === "plugins.entries.bad-plugin.config" && + issue.message.includes("invalid config"), + ); + expect(hasIssue).toBe(true); + } }); it("accepts known plugin ids", async () => { - await withTempHome(async (home) => { - const res = validateInHome(home, { - agents: { list: [{ id: "pi" }] }, - plugins: { enabled: false, entries: { discord: { enabled: true } } }, - }); - expect(res.ok).toBe(true); + const home = await createCaseHome(); + const res = validateInHome(home, { + agents: { list: [{ id: "pi" }] }, + plugins: { enabled: false, entries: { discord: { enabled: true } } }, }); + expect(res.ok).toBe(true); }); it("accepts plugin heartbeat targets", async () => { - await withTempHome(async (home) => { - const pluginDir = path.join(home, "bluebubbles-plugin"); - await writePluginFixture({ - dir: pluginDir, - id: "bluebubbles-plugin", - channels: ["bluebubbles"], - schema: { type: "object" }, - }); - - const res = validateInHome(home, { - agents: { defaults: { heartbeat: { target: "bluebubbles" } }, list: [{ id: "pi" }] }, - plugins: { enabled: false, load: { paths: [pluginDir] } }, - }); - expect(res.ok).toBe(true); + const home = await createCaseHome(); + const pluginDir = path.join(home, "bluebubbles-plugin"); + await writePluginFixture({ + dir: pluginDir, + id: "bluebubbles-plugin", + channels: ["bluebubbles"], + schema: { type: "object" }, }); + + const res = validateInHome(home, { + agents: { defaults: { heartbeat: { target: "bluebubbles" } }, list: [{ id: "pi" }] }, + plugins: { enabled: false, load: { paths: [pluginDir] } }, + }); + expect(res.ok).toBe(true); }); it("rejects unknown heartbeat targets", async () => { - await withTempHome(async (home) => { - const res = validateInHome(home, { - agents: { defaults: { heartbeat: { target: "not-a-channel" } }, list: [{ id: "pi" }] }, - }); - expect(res.ok).toBe(false); - if (!res.ok) { - expect(res.issues).toContainEqual({ - path: "agents.defaults.heartbeat.target", - message: "unknown heartbeat target: not-a-channel", - }); - } + const home = await createCaseHome(); + const res = validateInHome(home, { + agents: { defaults: { heartbeat: { target: "not-a-channel" } }, list: [{ id: "pi" }] }, }); + expect(res.ok).toBe(false); + if (!res.ok) { + expect(res.issues).toContainEqual({ + path: "agents.defaults.heartbeat.target", + message: "unknown heartbeat target: not-a-channel", + }); + } }); }); diff --git a/src/config/config.ts b/src/config/config.ts index 4761b7b215d..a20d9495b00 100644 --- a/src/config/config.ts +++ b/src/config/config.ts @@ -1,8 +1,10 @@ export { + clearConfigCache, createConfigIO, loadConfig, parseConfigJson5, readConfigFileSnapshot, + readConfigFileSnapshotForWrite, resolveConfigSnapshotHash, writeConfigFile, } from "./io.js"; diff --git a/src/config/env-preserve-io.test.ts b/src/config/env-preserve-io.test.ts new file mode 100644 index 00000000000..5e2d29a31bf --- /dev/null +++ b/src/config/env-preserve-io.test.ts @@ -0,0 +1,172 @@ +import fs from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { describe, it, expect } from "vitest"; +import { + createConfigIO, + readConfigFileSnapshotForWrite, + writeConfigFile as writeConfigFileViaWrapper, +} from "./io.js"; + +async function withTempConfig( + configContent: string, + run: (configPath: string) => Promise, +): Promise { + const dir = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-env-io-")); + const configPath = path.join(dir, "openclaw.json"); + await fs.writeFile(configPath, configContent); + try { + await run(configPath); + } finally { + await fs.rm(dir, { recursive: true, force: true }); + } +} + +async function withEnvOverrides( + updates: Record, + run: () => Promise, +): Promise { + const previous = new Map(); + for (const key of Object.keys(updates)) { + previous.set(key, process.env[key]); + } + + try { + for (const [key, value] of Object.entries(updates)) { + if (value === undefined) { + delete process.env[key]; + } else { + process.env[key] = value; + } + } + await run(); + } finally { + for (const [key, value] of previous.entries()) { + if (value === undefined) { + delete process.env[key]; + } else { + process.env[key] = value; + } + } + } +} + +describe("env snapshot TOCTOU via createConfigIO", () => { + it("restores env refs using read-time env even after env mutation", async () => { + const env: Record = { + MY_API_KEY: "original-key-123", + }; + + const configJson = JSON.stringify({ gateway: { remote: { token: "${MY_API_KEY}" } } }, null, 2); + + await withTempConfig(configJson, async (configPath) => { + // Instance A: read config (captures env snapshot) + const ioA = createConfigIO({ configPath, env: env as unknown as NodeJS.ProcessEnv }); + const firstRead = await ioA.readConfigFileSnapshotForWrite(); + expect(firstRead.snapshot.config.gateway?.remote?.token).toBe("original-key-123"); + + // Mutate env between read and write + env.MY_API_KEY = "mutated-key-456"; + + // Instance B: write config using explicit read context from A + const ioB = createConfigIO({ configPath, env: env as unknown as NodeJS.ProcessEnv }); + + // Write the resolved config back β€” should restore ${MY_API_KEY} + await ioB.writeConfigFile(firstRead.snapshot.config, firstRead.writeOptions); + + // Verify the written file still has ${MY_API_KEY}, not the resolved value + const written = await fs.readFile(configPath, "utf-8"); + const parsed = JSON.parse(written); + expect(parsed.gateway.remote.token).toBe("${MY_API_KEY}"); + }); + }); + + it("without snapshot bridging, mutated env causes incorrect restoration", async () => { + const env: Record = { + MY_API_KEY: "original-key-123", + }; + + const configJson = JSON.stringify({ gateway: { remote: { token: "${MY_API_KEY}" } } }, null, 2); + + await withTempConfig(configJson, async (configPath) => { + // Instance A: read config + const ioA = createConfigIO({ configPath, env: env as unknown as NodeJS.ProcessEnv }); + const snapshot = await ioA.readConfigFileSnapshot(); + + // Mutate env + env.MY_API_KEY = "mutated-key-456"; + + // Instance B: write WITHOUT snapshot bridging (simulates the old bug) + const ioB = createConfigIO({ configPath, env: env as unknown as NodeJS.ProcessEnv }); + // No explicit writeOptions β€” ioB uses live env + + await ioB.writeConfigFile(snapshot.config); + + // The written file should have the raw value because the live env + // no longer matches β€” restoreEnvVarRefs won't find a match + const written = await fs.readFile(configPath, "utf-8"); + const parsed = JSON.parse(written); + // Without snapshot, the resolved value "original-key-123" doesn't match + // live env "mutated-key-456", so restoration fails β€” value is written as-is + expect(parsed.gateway.remote.token).toBe("original-key-123"); + }); + }); +}); + +describe("env snapshot TOCTOU via wrapper APIs", () => { + it("uses explicit read context even if another read interleaves", async () => { + const configJson = JSON.stringify({ gateway: { remote: { token: "${MY_API_KEY}" } } }, null, 2); + await withTempConfig(configJson, async (configPath) => { + await withEnvOverrides( + { + OPENCLAW_CONFIG_PATH: configPath, + OPENCLAW_DISABLE_CONFIG_CACHE: "1", + MY_API_KEY: "original-key-123", + }, + async () => { + const firstRead = await readConfigFileSnapshotForWrite(); + expect(firstRead.snapshot.config.gateway?.remote?.token).toBe("original-key-123"); + + // Interleaving read from another request context with a different env value. + process.env.MY_API_KEY = "mutated-key-456"; + const secondRead = await readConfigFileSnapshotForWrite(); + expect(secondRead.snapshot.config.gateway?.remote?.token).toBe("mutated-key-456"); + + // Write using the first read's explicit context. + await writeConfigFileViaWrapper(firstRead.snapshot.config, firstRead.writeOptions); + const written = await fs.readFile(configPath, "utf-8"); + const parsed = JSON.parse(written); + expect(parsed.gateway.remote.token).toBe("${MY_API_KEY}"); + }, + ); + }); + }); + + it("ignores read context when expected config path does not match", async () => { + const configJson = JSON.stringify({ gateway: { remote: { token: "${MY_API_KEY}" } } }, null, 2); + await withTempConfig(configJson, async (configPath) => { + await withEnvOverrides( + { + OPENCLAW_CONFIG_PATH: configPath, + OPENCLAW_DISABLE_CONFIG_CACHE: "1", + MY_API_KEY: "original-key-123", + }, + async () => { + const firstRead = await readConfigFileSnapshotForWrite(); + expect(firstRead.snapshot.config.gateway?.remote?.token).toBe("original-key-123"); + expect(firstRead.writeOptions.expectedConfigPath).toBe(configPath); + + process.env.MY_API_KEY = "mutated-key-456"; + await writeConfigFileViaWrapper(firstRead.snapshot.config, { + ...firstRead.writeOptions, + expectedConfigPath: `${configPath}.different`, + }); + + const written = await fs.readFile(configPath, "utf-8"); + const parsed = JSON.parse(written); + expect(parsed.gateway.remote.token).toBe("original-key-123"); + }, + ); + }); + }); +}); diff --git a/src/config/env-preserve.test.ts b/src/config/env-preserve.test.ts new file mode 100644 index 00000000000..a3f9d63e4ae --- /dev/null +++ b/src/config/env-preserve.test.ts @@ -0,0 +1,182 @@ +import { describe, it, expect } from "vitest"; +import { restoreEnvVarRefs } from "./env-preserve.js"; + +describe("restoreEnvVarRefs", () => { + const env = { + ANTHROPIC_API_KEY: "sk-ant-api03-real-key", + OPENAI_API_KEY: "sk-openai-real-key", + MY_TOKEN: "tok-12345", + } as unknown as NodeJS.ProcessEnv; + + it("restores a simple ${VAR} reference when value matches", () => { + const incoming = { apiKey: "sk-ant-api03-real-key" }; + const parsed = { apiKey: "${ANTHROPIC_API_KEY}" }; + const result = restoreEnvVarRefs(incoming, parsed, env); + expect(result).toEqual({ apiKey: "${ANTHROPIC_API_KEY}" }); + }); + + it("keeps new value when caller intentionally changed it", () => { + const incoming = { apiKey: "sk-ant-new-different-key" }; + const parsed = { apiKey: "${ANTHROPIC_API_KEY}" }; + const result = restoreEnvVarRefs(incoming, parsed, env); + expect(result).toEqual({ apiKey: "sk-ant-new-different-key" }); + }); + + it("handles nested objects", () => { + const incoming = { + models: { + providers: { + anthropic: { apiKey: "sk-ant-api03-real-key" }, + openai: { apiKey: "sk-openai-real-key" }, + }, + }, + }; + const parsed = { + models: { + providers: { + anthropic: { apiKey: "${ANTHROPIC_API_KEY}" }, + openai: { apiKey: "${OPENAI_API_KEY}" }, + }, + }, + }; + const result = restoreEnvVarRefs(incoming, parsed, env); + expect(result).toEqual({ + models: { + providers: { + anthropic: { apiKey: "${ANTHROPIC_API_KEY}" }, + openai: { apiKey: "${OPENAI_API_KEY}" }, + }, + }, + }); + }); + + it("preserves new keys not in parsed", () => { + const incoming = { apiKey: "sk-ant-api03-real-key", newField: "hello" }; + const parsed = { apiKey: "${ANTHROPIC_API_KEY}" }; + const result = restoreEnvVarRefs(incoming, parsed, env); + expect(result).toEqual({ apiKey: "${ANTHROPIC_API_KEY}", newField: "hello" }); + }); + + it("handles non-env-var strings (no restoration needed)", () => { + const incoming = { name: "my-config" }; + const parsed = { name: "my-config" }; + const result = restoreEnvVarRefs(incoming, parsed, env); + expect(result).toEqual({ name: "my-config" }); + }); + + it("handles arrays", () => { + const incoming = ["sk-ant-api03-real-key", "literal"]; + const parsed = ["${ANTHROPIC_API_KEY}", "literal"]; + const result = restoreEnvVarRefs(incoming, parsed, env); + expect(result).toEqual(["${ANTHROPIC_API_KEY}", "literal"]); + }); + + it("handles null/undefined parsed gracefully", () => { + const incoming = { apiKey: "sk-ant-api03-real-key" }; + expect(restoreEnvVarRefs(incoming, null, env)).toEqual(incoming); + expect(restoreEnvVarRefs(incoming, undefined, env)).toEqual(incoming); + }); + + it("handles missing env var (cannot verify match)", () => { + const envMissing = {} as unknown as NodeJS.ProcessEnv; + const incoming = { apiKey: "some-value" }; + const parsed = { apiKey: "${MISSING_VAR}" }; + // Can't resolve the template, so keep incoming as-is + const result = restoreEnvVarRefs(incoming, parsed, envMissing); + expect(result).toEqual({ apiKey: "some-value" }); + }); + + it("handles composite template strings like prefix-${VAR}-suffix", () => { + const incoming = { url: "https://tok-12345.example.com" }; + const parsed = { url: "https://${MY_TOKEN}.example.com" }; + const result = restoreEnvVarRefs(incoming, parsed, env); + expect(result).toEqual({ url: "https://${MY_TOKEN}.example.com" }); + }); + + it("handles type mismatches between incoming and parsed", () => { + // Caller changed type from string to number + const incoming = { port: 8080 }; + const parsed = { port: "8080" }; + const result = restoreEnvVarRefs(incoming, parsed, env); + expect(result).toEqual({ port: 8080 }); + }); + + it("does not restore when parsed value has no env var pattern", () => { + const incoming = { apiKey: "sk-ant-api03-real-key" }; + const parsed = { apiKey: "sk-ant-api03-real-key" }; + const result = restoreEnvVarRefs(incoming, parsed, env); + expect(result).toEqual({ apiKey: "sk-ant-api03-real-key" }); + }); + + // Edge case: env mutation between read and write (Greptile comment #1) + // Scenario: config.env sets FOO=bar, which gets applied to process.env during loadConfig. + // Later writeConfigFile runs β€” the env has changed since the original read. + it("does not incorrectly restore when env var value changed between read and write", () => { + // At read time, MY_VAR was "original-value" and resolved ${MY_VAR} β†’ "original-value" + // Then config.env or external mutation changed MY_VAR to "mutated-value" + // Caller is writing back "original-value" (the value they got from the read) + const mutatedEnv = { MY_VAR: "mutated-value" } as unknown as NodeJS.ProcessEnv; + const incoming = { key: "original-value" }; + const parsed = { key: "${MY_VAR}" }; + + const result = restoreEnvVarRefs(incoming, parsed, mutatedEnv); + // Should NOT restore ${MY_VAR} because resolving it now gives "mutated-value", + // which doesn't match "original-value" β€” the caller's value should be kept + expect(result).toEqual({ key: "original-value" }); + }); + + it("correctly restores when env var value hasn't changed", () => { + const stableEnv = { MY_VAR: "stable-value" } as unknown as NodeJS.ProcessEnv; + const incoming = { key: "stable-value" }; + const parsed = { key: "${MY_VAR}" }; + + const result = restoreEnvVarRefs(incoming, parsed, stableEnv); + // Env value matches incoming β€” safe to restore + expect(result).toEqual({ key: "${MY_VAR}" }); + }); + + it("does not restore when env snapshot differs from live env (TOCTOU fix)", () => { + // With env snapshots: at read time MY_VAR was "old-value", so incoming is "old-value". + // Caller changed it to "new-value". Live env also changed to "new-value". + // But using the READ-TIME snapshot ("old-value"), we correctly see mismatch and keep incoming. + const readTimeEnv = { MY_VAR: "old-value" } as unknown as NodeJS.ProcessEnv; + const incoming = { key: "new-value" }; // caller intentionally changed this + const parsed = { key: "${MY_VAR}" }; + + const result = restoreEnvVarRefs(incoming, parsed, readTimeEnv); + // Using read-time snapshot: ${MY_VAR} resolves to "old-value", doesn't match "new-value" + // β†’ correctly keeps caller's new value + expect(result).toEqual({ key: "new-value" }); + }); + + // Edge case: $${VAR} escape sequence (Greptile comment #2) + it("handles $${VAR} escape sequence (literal ${VAR} in output)", () => { + // In the config file: $${ANTHROPIC_API_KEY} + // substituteString resolves this to literal "${ANTHROPIC_API_KEY}" + // So incoming would be "${ANTHROPIC_API_KEY}" (the literal text) + const incoming = { note: "${ANTHROPIC_API_KEY}" }; + const parsed = { note: "$${ANTHROPIC_API_KEY}" }; + + const result = restoreEnvVarRefs(incoming, parsed, env); + // Should restore the $${} escape, not try to resolve ${} inside it + expect(result).toEqual({ note: "$${ANTHROPIC_API_KEY}" }); + }); + + it("does not confuse $${VAR} escape with ${VAR} substitution", () => { + // Config has both: an escaped ref and a real ref + const incoming = { + literal: "${MY_TOKEN}", // from $${MY_TOKEN} β†’ literal "${MY_TOKEN}" + resolved: "tok-12345", // from ${MY_TOKEN} β†’ "tok-12345" + }; + const parsed = { + literal: "$${MY_TOKEN}", // escape sequence + resolved: "${MY_TOKEN}", // real env var ref + }; + + const result = restoreEnvVarRefs(incoming, parsed, env); + expect(result).toEqual({ + literal: "$${MY_TOKEN}", // should restore escape + resolved: "${MY_TOKEN}", // should restore ref + }); + }); +}); diff --git a/src/config/env-preserve.ts b/src/config/env-preserve.ts new file mode 100644 index 00000000000..a882357b9ec --- /dev/null +++ b/src/config/env-preserve.ts @@ -0,0 +1,141 @@ +/** + * Preserves `${VAR}` environment variable references during config write-back. + * + * When config is read, `${VAR}` references are resolved to their values. + * When writing back, callers pass the resolved config. This module detects + * values that match what a `${VAR}` reference would resolve to and restores + * the original reference, so env var references survive config round-trips. + * + * A value is restored only if: + * 1. The pre-substitution value contained a `${VAR}` pattern + * 2. Resolving that pattern with current env vars produces the incoming value + * + * If a caller intentionally set a new value (different from what the env var + * resolves to), the new value is kept as-is. + */ + +const ENV_VAR_PATTERN = /\$\{[A-Z_][A-Z0-9_]*\}/; + +function isPlainObject(value: unknown): value is Record { + return ( + typeof value === "object" && + value !== null && + !Array.isArray(value) && + Object.prototype.toString.call(value) === "[object Object]" + ); +} + +/** + * Check if a string contains any `${VAR}` env var references. + */ +function hasEnvVarRef(value: string): boolean { + return ENV_VAR_PATTERN.test(value); +} + +/** + * Resolve `${VAR}` references in a single string using the given env. + * Returns null if any referenced var is missing (instead of throwing). + * + * Mirrors the substitution semantics of `substituteString` in env-substitution.ts: + * - `${VAR}` β†’ env value (returns null if missing) + * - `$${VAR}` β†’ literal `${VAR}` (escape sequence) + */ +function tryResolveString(template: string, env: NodeJS.ProcessEnv): string | null { + const ENV_VAR_NAME = /^[A-Z_][A-Z0-9_]*$/; + const chunks: string[] = []; + + for (let i = 0; i < template.length; i++) { + if (template[i] === "$") { + // Escaped: $${VAR} -> literal ${VAR} + if (template[i + 1] === "$" && template[i + 2] === "{") { + const start = i + 3; + const end = template.indexOf("}", start); + if (end !== -1) { + const name = template.slice(start, end); + if (ENV_VAR_NAME.test(name)) { + chunks.push(`\${${name}}`); + i = end; + continue; + } + } + } + + // Substitution: ${VAR} -> env value + if (template[i + 1] === "{") { + const start = i + 2; + const end = template.indexOf("}", start); + if (end !== -1) { + const name = template.slice(start, end); + if (ENV_VAR_NAME.test(name)) { + const val = env[name]; + if (val === undefined || val === "") { + return null; + } + chunks.push(val); + i = end; + continue; + } + } + } + } + chunks.push(template[i]); + } + + return chunks.join(""); +} + +/** + * Deep-walk the incoming config and restore `${VAR}` references from the + * pre-substitution parsed config wherever the resolved value matches. + * + * @param incoming - The resolved config about to be written + * @param parsed - The pre-substitution parsed config (from the current file on disk) + * @param env - Environment variables for verification + * @returns A new config object with env var references restored where appropriate + */ +export function restoreEnvVarRefs( + incoming: unknown, + parsed: unknown, + env: NodeJS.ProcessEnv = process.env, +): unknown { + // If parsed has no env var refs at this level, return incoming as-is + if (parsed === null || parsed === undefined) { + return incoming; + } + + // String leaf: check if parsed was a ${VAR} template that resolves to incoming + if (typeof incoming === "string" && typeof parsed === "string") { + if (hasEnvVarRef(parsed)) { + const resolved = tryResolveString(parsed, env); + if (resolved === incoming) { + // The incoming value matches what the env var resolves to β€” restore the reference + return parsed; + } + } + return incoming; + } + + // Arrays: walk element by element + if (Array.isArray(incoming) && Array.isArray(parsed)) { + return incoming.map((item, i) => + i < parsed.length ? restoreEnvVarRefs(item, parsed[i], env) : item, + ); + } + + // Objects: walk key by key + if (isPlainObject(incoming) && isPlainObject(parsed)) { + const result: Record = {}; + for (const [key, value] of Object.entries(incoming)) { + if (key in parsed) { + result[key] = restoreEnvVarRefs(value, parsed[key], env); + } else { + // New key added by caller β€” keep as-is + result[key] = value; + } + } + return result; + } + + // Mismatched types or primitives β€” keep incoming + return incoming; +} diff --git a/src/config/io.ts b/src/config/io.ts index 184f73942aa..a2d0b4c791e 100644 --- a/src/config/io.ts +++ b/src/config/io.ts @@ -25,6 +25,7 @@ import { applySessionDefaults, applyTalkApiKey, } from "./defaults.js"; +import { restoreEnvVarRefs } from "./env-preserve.js"; import { MissingEnvVarError, containsEnvVarReference, @@ -67,9 +68,58 @@ const SHELL_ENV_EXPECTED_KEYS = [ ]; const CONFIG_BACKUP_COUNT = 5; +const CONFIG_AUDIT_LOG_FILENAME = "config-audit.jsonl"; const loggedInvalidConfigs = new Set(); +type ConfigWriteAuditResult = "rename" | "copy-fallback" | "failed"; + +type ConfigWriteAuditRecord = { + ts: string; + source: "config-io"; + event: "config.write"; + result: ConfigWriteAuditResult; + configPath: string; + pid: number; + ppid: number; + cwd: string; + argv: string[]; + execArgv: string[]; + watchMode: boolean; + watchSession: string | null; + watchCommand: string | null; + existsBefore: boolean; + previousHash: string | null; + nextHash: string | null; + previousBytes: number | null; + nextBytes: number | null; + changedPathCount: number | null; + hasMetaBefore: boolean; + hasMetaAfter: boolean; + gatewayModeBefore: string | null; + gatewayModeAfter: string | null; + suspicious: string[]; + errorCode?: string; + errorMessage?: string; +}; + export type ParseConfigJson5Result = { ok: true; parsed: unknown } | { ok: false; error: string }; +export type ConfigWriteOptions = { + /** + * Read-time env snapshot used to validate `${VAR}` restoration decisions. + * If omitted, write falls back to current process env. + */ + envSnapshotForRestore?: Record; + /** + * Optional safety check: only use envSnapshotForRestore when writing the + * same config file path that produced the snapshot. + */ + expectedConfigPath?: string; +}; + +export type ReadConfigFileSnapshotForWriteResult = { + snapshot: ConfigFileSnapshot; + writeOptions: ConfigWriteOptions; +}; function hashConfigRaw(raw: string | null): string { return crypto @@ -105,6 +155,26 @@ function isPlainObject(value: unknown): value is Record { return typeof value === "object" && value !== null && !Array.isArray(value); } +function hasConfigMeta(value: unknown): boolean { + if (!isPlainObject(value)) { + return false; + } + const meta = value.meta; + return isPlainObject(meta); +} + +function resolveGatewayMode(value: unknown): string | null { + if (!isPlainObject(value)) { + return null; + } + const gateway = value.gateway; + if (!isPlainObject(gateway) || typeof gateway.mode !== "string") { + return null; + } + const trimmed = gateway.mode.trim(); + return trimmed.length > 0 ? trimmed : null; +} + function cloneUnknown(value: T): T { return structuredClone(value); } @@ -289,6 +359,55 @@ async function rotateConfigBackups(configPath: string, ioFs: typeof fs.promises) }); } +function resolveConfigAuditLogPath(env: NodeJS.ProcessEnv, homedir: () => string): string { + return path.join(resolveStateDir(env, homedir), "logs", CONFIG_AUDIT_LOG_FILENAME); +} + +function resolveConfigWriteSuspiciousReasons(params: { + existsBefore: boolean; + previousBytes: number | null; + nextBytes: number | null; + hasMetaBefore: boolean; + gatewayModeBefore: string | null; + gatewayModeAfter: string | null; +}): string[] { + const reasons: string[] = []; + if (!params.existsBefore) { + return reasons; + } + if ( + typeof params.previousBytes === "number" && + typeof params.nextBytes === "number" && + params.previousBytes >= 512 && + params.nextBytes < Math.floor(params.previousBytes * 0.5) + ) { + reasons.push(`size-drop:${params.previousBytes}->${params.nextBytes}`); + } + if (!params.hasMetaBefore) { + reasons.push("missing-meta-before-write"); + } + if (params.gatewayModeBefore && !params.gatewayModeAfter) { + reasons.push("gateway-mode-removed"); + } + return reasons; +} + +async function appendConfigWriteAuditRecord( + deps: Required, + record: ConfigWriteAuditRecord, +): Promise { + try { + const auditPath = resolveConfigAuditLogPath(deps.env, deps.homedir); + await deps.fs.promises.mkdir(path.dirname(auditPath), { recursive: true, mode: 0o700 }); + await deps.fs.promises.appendFile(auditPath, `${JSON.stringify(record)}\n`, { + encoding: "utf-8", + mode: 0o600, + }); + } catch { + // best-effort + } +} + export type ConfigIoDeps = { fs?: typeof fs; json5?: typeof JSON5; @@ -390,6 +509,43 @@ export function parseConfigJson5( } } +type ConfigReadResolution = { + resolvedConfigRaw: unknown; + envSnapshotForRestore: Record; +}; + +function resolveConfigIncludesForRead( + parsed: unknown, + configPath: string, + deps: Required, +): unknown { + return resolveConfigIncludes(parsed, configPath, { + readFile: (candidate) => deps.fs.readFileSync(candidate, "utf-8"), + parseJson: (raw) => deps.json5.parse(raw), + }); +} + +function resolveConfigForRead( + resolvedIncludes: unknown, + env: NodeJS.ProcessEnv, +): ConfigReadResolution { + // Apply config.env to process.env BEFORE substitution so ${VAR} can reference config-defined vars. + if (resolvedIncludes && typeof resolvedIncludes === "object" && "env" in resolvedIncludes) { + applyConfigEnv(resolvedIncludes as OpenClawConfig, env); + } + + return { + resolvedConfigRaw: resolveConfigEnvVars(resolvedIncludes, env), + // Capture env snapshot after substitution for write-time ${VAR} restoration. + envSnapshotForRestore: { ...env } as Record, + }; +} + +type ReadConfigFileSnapshotInternalResult = { + snapshot: ConfigFileSnapshot; + envSnapshotForRestore?: Record; +}; + export function createConfigIO(overrides: ConfigIoDeps = {}) { const deps = normalizeDeps(overrides); const requestedConfigPath = resolveConfigPathForDeps(deps); @@ -416,22 +572,10 @@ export function createConfigIO(overrides: ConfigIoDeps = {}) { } const raw = deps.fs.readFileSync(configPath, "utf-8"); const parsed = deps.json5.parse(raw); - - // Resolve $include directives before validation - const resolved = resolveConfigIncludes(parsed, configPath, { - readFile: (p) => deps.fs.readFileSync(p, "utf-8"), - parseJson: (raw) => deps.json5.parse(raw), - }); - - // Apply config.env to process.env BEFORE substitution so ${VAR} can reference config-defined vars - if (resolved && typeof resolved === "object" && "env" in resolved) { - applyConfigEnv(resolved as OpenClawConfig, deps.env); - } - - // Substitute ${VAR} env var references - const substituted = resolveConfigEnvVars(resolved, deps.env); - - const resolvedConfig = substituted; + const { resolvedConfigRaw: resolvedConfig } = resolveConfigForRead( + resolveConfigIncludesForRead(parsed, configPath, deps), + deps.env, + ); warnOnConfigMiskeys(resolvedConfig, deps.logger); if (typeof resolvedConfig !== "object" || resolvedConfig === null) { return {}; @@ -511,7 +655,7 @@ export function createConfigIO(overrides: ConfigIoDeps = {}) { } } - async function readConfigFileSnapshot(): Promise { + async function readConfigFileSnapshotInternal(): Promise { maybeLoadDotEnvForConfig(deps.env); const exists = deps.fs.existsSync(configPath); if (!exists) { @@ -527,17 +671,19 @@ export function createConfigIO(overrides: ConfigIoDeps = {}) { ); const legacyIssues: LegacyConfigIssue[] = []; return { - path: configPath, - exists: false, - raw: null, - parsed: {}, - resolved: {}, - valid: true, - config, - hash, - issues: [], - warnings: [], - legacyIssues, + snapshot: { + path: configPath, + exists: false, + raw: null, + parsed: {}, + resolved: {}, + valid: true, + config, + hash, + issues: [], + warnings: [], + legacyIssues, + }, }; } @@ -547,141 +693,163 @@ export function createConfigIO(overrides: ConfigIoDeps = {}) { const parsedRes = parseConfigJson5(raw, deps.json5); if (!parsedRes.ok) { return { - path: configPath, - exists: true, - raw, - parsed: {}, - resolved: {}, - valid: false, - config: {}, - hash, - issues: [{ path: "", message: `JSON5 parse failed: ${parsedRes.error}` }], - warnings: [], - legacyIssues: [], + snapshot: { + path: configPath, + exists: true, + raw, + parsed: {}, + resolved: {}, + valid: false, + config: {}, + hash, + issues: [{ path: "", message: `JSON5 parse failed: ${parsedRes.error}` }], + warnings: [], + legacyIssues: [], + }, }; } // Resolve $include directives let resolved: unknown; try { - resolved = resolveConfigIncludes(parsedRes.parsed, configPath, { - readFile: (p) => deps.fs.readFileSync(p, "utf-8"), - parseJson: (raw) => deps.json5.parse(raw), - }); + resolved = resolveConfigIncludesForRead(parsedRes.parsed, configPath, deps); } catch (err) { const message = err instanceof ConfigIncludeError ? err.message : `Include resolution failed: ${String(err)}`; return { - path: configPath, - exists: true, - raw, - parsed: parsedRes.parsed, - resolved: coerceConfig(parsedRes.parsed), - valid: false, - config: coerceConfig(parsedRes.parsed), - hash, - issues: [{ path: "", message }], - warnings: [], - legacyIssues: [], + snapshot: { + path: configPath, + exists: true, + raw, + parsed: parsedRes.parsed, + resolved: coerceConfig(parsedRes.parsed), + valid: false, + config: coerceConfig(parsedRes.parsed), + hash, + issues: [{ path: "", message }], + warnings: [], + legacyIssues: [], + }, }; } - // Apply config.env to process.env BEFORE substitution so ${VAR} can reference config-defined vars - if (resolved && typeof resolved === "object" && "env" in resolved) { - applyConfigEnv(resolved as OpenClawConfig, deps.env); - } - - // Substitute ${VAR} env var references - let substituted: unknown; + let readResolution: ConfigReadResolution; try { - substituted = resolveConfigEnvVars(resolved, deps.env); + readResolution = resolveConfigForRead(resolved, deps.env); } catch (err) { const message = err instanceof MissingEnvVarError ? err.message : `Env var substitution failed: ${String(err)}`; return { - path: configPath, - exists: true, - raw, - parsed: parsedRes.parsed, - resolved: coerceConfig(resolved), - valid: false, - config: coerceConfig(resolved), - hash, - issues: [{ path: "", message }], - warnings: [], - legacyIssues: [], + snapshot: { + path: configPath, + exists: true, + raw, + parsed: parsedRes.parsed, + resolved: coerceConfig(resolved), + valid: false, + config: coerceConfig(resolved), + hash, + issues: [{ path: "", message }], + warnings: [], + legacyIssues: [], + }, }; } - const resolvedConfigRaw = substituted; + const resolvedConfigRaw = readResolution.resolvedConfigRaw; const legacyIssues = findLegacyConfigIssues(resolvedConfigRaw); const validated = validateConfigObjectWithPlugins(resolvedConfigRaw); if (!validated.ok) { return { - path: configPath, - exists: true, - raw, - parsed: parsedRes.parsed, - resolved: coerceConfig(resolvedConfigRaw), - valid: false, - config: coerceConfig(resolvedConfigRaw), - hash, - issues: validated.issues, - warnings: validated.warnings, - legacyIssues, + snapshot: { + path: configPath, + exists: true, + raw, + parsed: parsedRes.parsed, + resolved: coerceConfig(resolvedConfigRaw), + valid: false, + config: coerceConfig(resolvedConfigRaw), + hash, + issues: validated.issues, + warnings: validated.warnings, + legacyIssues, + }, }; } warnIfConfigFromFuture(validated.config, deps.logger); return { - path: configPath, - exists: true, - raw, - parsed: parsedRes.parsed, - // Use resolvedConfigRaw (after $include and ${ENV} substitution but BEFORE runtime defaults) - // for config set/unset operations (issue #6070) - resolved: coerceConfig(resolvedConfigRaw), - valid: true, - config: normalizeConfigPaths( - applyTalkApiKey( - applyModelDefaults( - applyAgentDefaults( - applySessionDefaults(applyLoggingDefaults(applyMessageDefaults(validated.config))), + snapshot: { + path: configPath, + exists: true, + raw, + parsed: parsedRes.parsed, + // Use resolvedConfigRaw (after $include and ${ENV} substitution but BEFORE runtime defaults) + // for config set/unset operations (issue #6070) + resolved: coerceConfig(resolvedConfigRaw), + valid: true, + config: normalizeConfigPaths( + applyTalkApiKey( + applyModelDefaults( + applyAgentDefaults( + applySessionDefaults( + applyLoggingDefaults(applyMessageDefaults(validated.config)), + ), + ), ), ), ), - ), - hash, - issues: [], - warnings: validated.warnings, - legacyIssues, + hash, + issues: [], + warnings: validated.warnings, + legacyIssues, + }, + envSnapshotForRestore: readResolution.envSnapshotForRestore, }; } catch (err) { return { - path: configPath, - exists: true, - raw: null, - parsed: {}, - resolved: {}, - valid: false, - config: {}, - hash: hashConfigRaw(null), - issues: [{ path: "", message: `read failed: ${String(err)}` }], - warnings: [], - legacyIssues: [], + snapshot: { + path: configPath, + exists: true, + raw: null, + parsed: {}, + resolved: {}, + valid: false, + config: {}, + hash: hashConfigRaw(null), + issues: [{ path: "", message: `read failed: ${String(err)}` }], + warnings: [], + legacyIssues: [], + }, }; } } - async function writeConfigFile(cfg: OpenClawConfig) { + async function readConfigFileSnapshot(): Promise { + const result = await readConfigFileSnapshotInternal(); + return result.snapshot; + } + + async function readConfigFileSnapshotForWrite(): Promise { + const result = await readConfigFileSnapshotInternal(); + return { + snapshot: result.snapshot, + writeOptions: { + envSnapshotForRestore: result.envSnapshotForRestore, + expectedConfigPath: configPath, + }, + }; + } + + async function writeConfigFile(cfg: OpenClawConfig, options: ConfigWriteOptions = {}) { clearConfigCache(); let persistCandidate: unknown = cfg; - const snapshot = await readConfigFileSnapshot(); + const { snapshot } = await readConfigFileSnapshotInternal(); let envRefMap: Map | null = null; let changedPaths: Set | null = null; if (snapshot.valid && snapshot.exists) { @@ -716,51 +884,184 @@ export function createConfigIO(overrides: ConfigIoDeps = {}) { .join("\n"); deps.logger.warn(`Config warnings:\n${details}`); } + + // Restore ${VAR} env var references that were resolved during config loading. + // Read the current file (pre-substitution) and restore any references whose + // resolved values match the incoming config β€” so we don't overwrite + // "${ANTHROPIC_API_KEY}" with "sk-ant-..." when the caller didn't change it. + // + // We use only the root file's parsed content (no $include resolution) to avoid + // pulling values from included files into the root config on write-back. + // Apply env restoration to validated.config (which has runtime defaults stripped + // per issue #6070) rather than the raw caller input. + let cfgToWrite = validated.config; + try { + if (deps.fs.existsSync(configPath)) { + const currentRaw = await deps.fs.promises.readFile(configPath, "utf-8"); + const parsedRes = parseConfigJson5(currentRaw, deps.json5); + if (parsedRes.ok) { + // Use env snapshot from when config was loaded (if available) to avoid + // TOCTOU issues where env changes between load and write. Falls back to + // live env if no snapshot exists (e.g., first write before any load). + const envForRestore = options.envSnapshotForRestore ?? deps.env; + cfgToWrite = restoreEnvVarRefs( + cfgToWrite, + parsedRes.parsed, + envForRestore, + ) as OpenClawConfig; + } + } + } catch { + // If reading the current file fails, write cfg as-is (no env restoration) + } + const dir = path.dirname(configPath); await deps.fs.promises.mkdir(dir, { recursive: true, mode: 0o700 }); const outputConfig = envRefMap && changedPaths - ? (restoreEnvRefsFromMap(validated.config, "", envRefMap, changedPaths) as OpenClawConfig) - : validated.config; + ? (restoreEnvRefsFromMap(cfgToWrite, "", envRefMap, changedPaths) as OpenClawConfig) + : cfgToWrite; // Do NOT apply runtime defaults when writing β€” user config should only contain // explicitly set values. Runtime defaults are applied when loading (issue #6070). - const json = JSON.stringify(stampConfigVersion(outputConfig), null, 2).trimEnd().concat("\n"); + const stampedOutputConfig = stampConfigVersion(outputConfig); + const json = JSON.stringify(stampedOutputConfig, null, 2).trimEnd().concat("\n"); + const nextHash = hashConfigRaw(json); + const previousHash = resolveConfigSnapshotHash(snapshot); + const changedPathCount = changedPaths?.size; + const previousBytes = + typeof snapshot.raw === "string" ? Buffer.byteLength(snapshot.raw, "utf-8") : null; + const nextBytes = Buffer.byteLength(json, "utf-8"); + const hasMetaBefore = hasConfigMeta(snapshot.parsed); + const hasMetaAfter = hasConfigMeta(stampedOutputConfig); + const gatewayModeBefore = resolveGatewayMode(snapshot.resolved); + const gatewayModeAfter = resolveGatewayMode(stampedOutputConfig); + const suspiciousReasons = resolveConfigWriteSuspiciousReasons({ + existsBefore: snapshot.exists, + previousBytes, + nextBytes, + hasMetaBefore, + gatewayModeBefore, + gatewayModeAfter, + }); + const logConfigOverwrite = () => { + if (!snapshot.exists) { + return; + } + const isVitest = deps.env.VITEST === "true"; + const shouldLogInVitest = deps.env.OPENCLAW_TEST_CONFIG_OVERWRITE_LOG === "1"; + if (isVitest && !shouldLogInVitest) { + return; + } + const changeSummary = + typeof changedPathCount === "number" ? `, changedPaths=${changedPathCount}` : ""; + deps.logger.warn( + `Config overwrite: ${configPath} (sha256 ${previousHash ?? "unknown"} -> ${nextHash}, backup=${configPath}.bak${changeSummary})`, + ); + }; + const logConfigWriteAnomalies = () => { + if (suspiciousReasons.length === 0) { + return; + } + deps.logger.warn(`Config write anomaly: ${configPath} (${suspiciousReasons.join(", ")})`); + }; + const auditRecordBase = { + ts: new Date().toISOString(), + source: "config-io" as const, + event: "config.write" as const, + configPath, + pid: process.pid, + ppid: process.ppid, + cwd: process.cwd(), + argv: process.argv.slice(0, 8), + execArgv: process.execArgv.slice(0, 8), + watchMode: deps.env.OPENCLAW_WATCH_MODE === "1", + watchSession: + typeof deps.env.OPENCLAW_WATCH_SESSION === "string" && + deps.env.OPENCLAW_WATCH_SESSION.trim().length > 0 + ? deps.env.OPENCLAW_WATCH_SESSION.trim() + : null, + watchCommand: + typeof deps.env.OPENCLAW_WATCH_COMMAND === "string" && + deps.env.OPENCLAW_WATCH_COMMAND.trim().length > 0 + ? deps.env.OPENCLAW_WATCH_COMMAND.trim() + : null, + existsBefore: snapshot.exists, + previousHash: previousHash ?? null, + nextHash, + previousBytes, + nextBytes, + changedPathCount: typeof changedPathCount === "number" ? changedPathCount : null, + hasMetaBefore, + hasMetaAfter, + gatewayModeBefore, + gatewayModeAfter, + suspicious: suspiciousReasons, + }; + const appendWriteAudit = async (result: ConfigWriteAuditResult, err?: unknown) => { + const errorCode = + err && typeof err === "object" && "code" in err && typeof err.code === "string" + ? err.code + : undefined; + const errorMessage = + err && typeof err === "object" && "message" in err && typeof err.message === "string" + ? err.message + : undefined; + await appendConfigWriteAuditRecord(deps, { + ...auditRecordBase, + result, + nextHash: result === "failed" ? null : auditRecordBase.nextHash, + nextBytes: result === "failed" ? null : auditRecordBase.nextBytes, + errorCode, + errorMessage, + }); + }; const tmp = path.join( dir, `${path.basename(configPath)}.${process.pid}.${crypto.randomUUID()}.tmp`, ); - await deps.fs.promises.writeFile(tmp, json, { - encoding: "utf-8", - mode: 0o600, - }); - - if (deps.fs.existsSync(configPath)) { - await rotateConfigBackups(configPath, deps.fs.promises); - await deps.fs.promises.copyFile(configPath, `${configPath}.bak`).catch(() => { - // best-effort - }); - } - try { - await deps.fs.promises.rename(tmp, configPath); - } catch (err) { - const code = (err as { code?: string }).code; - // Windows doesn't reliably support atomic replace via rename when dest exists. - if (code === "EPERM" || code === "EEXIST") { - await deps.fs.promises.copyFile(tmp, configPath); - await deps.fs.promises.chmod(configPath, 0o600).catch(() => { + await deps.fs.promises.writeFile(tmp, json, { + encoding: "utf-8", + mode: 0o600, + }); + + if (deps.fs.existsSync(configPath)) { + await rotateConfigBackups(configPath, deps.fs.promises); + await deps.fs.promises.copyFile(configPath, `${configPath}.bak`).catch(() => { // best-effort }); + } + + try { + await deps.fs.promises.rename(tmp, configPath); + } catch (err) { + const code = (err as { code?: string }).code; + // Windows doesn't reliably support atomic replace via rename when dest exists. + if (code === "EPERM" || code === "EEXIST") { + await deps.fs.promises.copyFile(tmp, configPath); + await deps.fs.promises.chmod(configPath, 0o600).catch(() => { + // best-effort + }); + await deps.fs.promises.unlink(tmp).catch(() => { + // best-effort + }); + logConfigOverwrite(); + logConfigWriteAnomalies(); + await appendWriteAudit("copy-fallback"); + return; + } await deps.fs.promises.unlink(tmp).catch(() => { // best-effort }); - return; + throw err; } - await deps.fs.promises.unlink(tmp).catch(() => { - // best-effort - }); + logConfigOverwrite(); + logConfigWriteAnomalies(); + await appendWriteAudit("rename"); + } catch (err) { + await appendWriteAudit("failed", err); throw err; } } @@ -769,6 +1070,7 @@ export function createConfigIO(overrides: ConfigIoDeps = {}) { configPath, loadConfig, readConfigFileSnapshot, + readConfigFileSnapshotForWrite, writeConfigFile, }; } @@ -805,7 +1107,7 @@ function shouldUseConfigCache(env: NodeJS.ProcessEnv): boolean { return resolveConfigCacheMs(env) > 0; } -function clearConfigCache(): void { +export function clearConfigCache(): void { configCache = null; } @@ -837,7 +1139,18 @@ export async function readConfigFileSnapshot(): Promise { return await createConfigIO().readConfigFileSnapshot(); } -export async function writeConfigFile(cfg: OpenClawConfig): Promise { - clearConfigCache(); - await createConfigIO().writeConfigFile(cfg); +export async function readConfigFileSnapshotForWrite(): Promise { + return await createConfigIO().readConfigFileSnapshotForWrite(); +} + +export async function writeConfigFile( + cfg: OpenClawConfig, + options: ConfigWriteOptions = {}, +): Promise { + const io = createConfigIO(); + const sameConfigPath = + options.expectedConfigPath === undefined || options.expectedConfigPath === io.configPath; + await io.writeConfigFile(cfg, { + envSnapshotForRestore: sameConfigPath ? options.envSnapshotForRestore : undefined, + }); } diff --git a/src/config/io.write-config.test.ts b/src/config/io.write-config.test.ts index 2aa85b20d46..59af3e99383 100644 --- a/src/config/io.write-config.test.ts +++ b/src/config/io.write-config.test.ts @@ -1,10 +1,82 @@ import fs from "node:fs/promises"; +import os from "node:os"; import path from "node:path"; -import { describe, expect, it } from "vitest"; +import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; import { createConfigIO } from "./io.js"; -import { withTempHome } from "./test-helpers.js"; + +type HomeEnvSnapshot = { + home: string | undefined; + userProfile: string | undefined; + homeDrive: string | undefined; + homePath: string | undefined; + stateDir: string | undefined; +}; + +function snapshotHomeEnv(): HomeEnvSnapshot { + return { + home: process.env.HOME, + userProfile: process.env.USERPROFILE, + homeDrive: process.env.HOMEDRIVE, + homePath: process.env.HOMEPATH, + stateDir: process.env.OPENCLAW_STATE_DIR, + }; +} + +function restoreHomeEnv(snapshot: HomeEnvSnapshot) { + const restoreKey = (key: string, value: string | undefined) => { + if (value === undefined) { + delete process.env[key]; + } else { + process.env[key] = value; + } + }; + restoreKey("HOME", snapshot.home); + restoreKey("USERPROFILE", snapshot.userProfile); + restoreKey("HOMEDRIVE", snapshot.homeDrive); + restoreKey("HOMEPATH", snapshot.homePath); + restoreKey("OPENCLAW_STATE_DIR", snapshot.stateDir); +} describe("config io write", () => { + let fixtureRoot = ""; + let fixtureCount = 0; + const silentLogger = { + warn: () => {}, + error: () => {}, + }; + + beforeAll(async () => { + fixtureRoot = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-config-io-")); + }); + + afterAll(async () => { + await fs.rm(fixtureRoot, { recursive: true, force: true }); + }); + + const withTempHome = async (fn: (home: string) => Promise): Promise => { + const home = path.join(fixtureRoot, `home-${fixtureCount++}`); + await fs.mkdir(path.join(home, ".openclaw"), { recursive: true }); + + const snapshot = snapshotHomeEnv(); + process.env.HOME = home; + process.env.USERPROFILE = home; + process.env.OPENCLAW_STATE_DIR = path.join(home, ".openclaw"); + + if (process.platform === "win32") { + const match = home.match(/^([A-Za-z]:)(.*)$/); + if (match) { + process.env.HOMEDRIVE = match[1]; + process.env.HOMEPATH = match[2] || "\\"; + } + } + + try { + return await fn(home); + } finally { + restoreHomeEnv(snapshot); + } + }; + it("persists caller changes onto resolved config without leaking runtime defaults", async () => { await withTempHome(async (home) => { const configPath = path.join(home, ".openclaw", "openclaw.json"); @@ -18,6 +90,7 @@ describe("config io write", () => { const io = createConfigIO({ env: {} as NodeJS.ProcessEnv, homedir: () => home, + logger: silentLogger, }); const snapshot = await io.readConfigFileSnapshot(); @@ -76,6 +149,7 @@ describe("config io write", () => { const io = createConfigIO({ env: { OPENAI_API_KEY: "sk-secret" } as NodeJS.ProcessEnv, homedir: () => home, + logger: silentLogger, }); const snapshot = await io.readConfigFileSnapshot(); @@ -131,6 +205,7 @@ describe("config io write", () => { const io = createConfigIO({ env: { DISCORD_USER_ID: "999" } as NodeJS.ProcessEnv, homedir: () => home, + logger: silentLogger, }); const snapshot = await io.readConfigFileSnapshot(); @@ -174,4 +249,153 @@ describe("config io write", () => { ]); }); }); + + it("logs an overwrite audit entry when replacing an existing config file", async () => { + await withTempHome(async (home) => { + const configPath = path.join(home, ".openclaw", "openclaw.json"); + await fs.mkdir(path.dirname(configPath), { recursive: true }); + await fs.writeFile( + configPath, + JSON.stringify({ gateway: { port: 18789 } }, null, 2), + "utf-8", + ); + const warn = vi.fn(); + const io = createConfigIO({ + env: {} as NodeJS.ProcessEnv, + homedir: () => home, + logger: { + warn, + error: vi.fn(), + }, + }); + + const snapshot = await io.readConfigFileSnapshot(); + expect(snapshot.valid).toBe(true); + const next = structuredClone(snapshot.config); + next.gateway = { + ...next.gateway, + auth: { mode: "token" }, + }; + + await io.writeConfigFile(next); + + const overwriteLog = warn.mock.calls + .map((call) => call[0]) + .find((entry) => typeof entry === "string" && entry.startsWith("Config overwrite:")); + expect(typeof overwriteLog).toBe("string"); + expect(overwriteLog).toContain(configPath); + expect(overwriteLog).toContain(`${configPath}.bak`); + expect(overwriteLog).toContain("sha256"); + }); + }); + + it("does not log an overwrite audit entry when creating config for the first time", async () => { + await withTempHome(async (home) => { + const warn = vi.fn(); + const io = createConfigIO({ + env: {} as NodeJS.ProcessEnv, + homedir: () => home, + logger: { + warn, + error: vi.fn(), + }, + }); + + await io.writeConfigFile({ + gateway: { mode: "local" }, + }); + + const overwriteLogs = warn.mock.calls.filter( + (call) => typeof call[0] === "string" && call[0].startsWith("Config overwrite:"), + ); + expect(overwriteLogs).toHaveLength(0); + }); + }); + + it("appends config write audit JSONL entries with forensic metadata", async () => { + await withTempHome(async (home) => { + const configPath = path.join(home, ".openclaw", "openclaw.json"); + const auditPath = path.join(home, ".openclaw", "logs", "config-audit.jsonl"); + await fs.mkdir(path.dirname(configPath), { recursive: true }); + await fs.writeFile( + configPath, + JSON.stringify({ gateway: { port: 18789 } }, null, 2), + "utf-8", + ); + + const io = createConfigIO({ + env: {} as NodeJS.ProcessEnv, + homedir: () => home, + logger: { + warn: vi.fn(), + error: vi.fn(), + }, + }); + + const snapshot = await io.readConfigFileSnapshot(); + expect(snapshot.valid).toBe(true); + + const next = structuredClone(snapshot.config); + next.gateway = { + ...next.gateway, + mode: "local", + }; + + await io.writeConfigFile(next); + + const lines = (await fs.readFile(auditPath, "utf-8")).trim().split("\n").filter(Boolean); + expect(lines.length).toBeGreaterThan(0); + const last = JSON.parse(lines.at(-1) ?? "{}") as Record; + expect(last.source).toBe("config-io"); + expect(last.event).toBe("config.write"); + expect(last.configPath).toBe(configPath); + expect(last.existsBefore).toBe(true); + expect(last.hasMetaAfter).toBe(true); + expect(last.previousHash).toBeTypeOf("string"); + expect(last.nextHash).toBeTypeOf("string"); + expect(last.result === "rename" || last.result === "copy-fallback").toBe(true); + }); + }); + + it("records gateway watch session markers in config audit entries", async () => { + await withTempHome(async (home) => { + const configPath = path.join(home, ".openclaw", "openclaw.json"); + const auditPath = path.join(home, ".openclaw", "logs", "config-audit.jsonl"); + await fs.mkdir(path.dirname(configPath), { recursive: true }); + await fs.writeFile( + configPath, + JSON.stringify({ gateway: { mode: "local" } }, null, 2), + "utf-8", + ); + + const io = createConfigIO({ + env: { + OPENCLAW_WATCH_MODE: "1", + OPENCLAW_WATCH_SESSION: "watch-session-1", + OPENCLAW_WATCH_COMMAND: "gateway --force", + } as NodeJS.ProcessEnv, + homedir: () => home, + logger: { + warn: vi.fn(), + error: vi.fn(), + }, + }); + + const snapshot = await io.readConfigFileSnapshot(); + expect(snapshot.valid).toBe(true); + const next = structuredClone(snapshot.config); + next.gateway = { + ...next.gateway, + bind: "loopback", + }; + + await io.writeConfigFile(next); + + const lines = (await fs.readFile(auditPath, "utf-8")).trim().split("\n").filter(Boolean); + const last = JSON.parse(lines.at(-1) ?? "{}") as Record; + expect(last.watchMode).toBe(true); + expect(last.watchSession).toBe("watch-session-1"); + expect(last.watchCommand).toBe("gateway --force"); + }); + }); }); diff --git a/src/config/sessions.ts b/src/config/sessions.ts index 20de39409b1..0ea031cf050 100644 --- a/src/config/sessions.ts +++ b/src/config/sessions.ts @@ -7,3 +7,4 @@ export * from "./sessions/session-key.js"; export * from "./sessions/store.js"; export * from "./sessions/types.js"; export * from "./sessions/transcript.js"; +export * from "./sessions/delivery-info.js"; diff --git a/src/config/sessions/delivery-info.ts b/src/config/sessions/delivery-info.ts new file mode 100644 index 00000000000..006f1db4490 --- /dev/null +++ b/src/config/sessions/delivery-info.ts @@ -0,0 +1,46 @@ +import { loadConfig } from "../io.js"; +import { resolveStorePath } from "./paths.js"; +import { loadSessionStore } from "./store.js"; + +/** + * Extract deliveryContext and threadId from a sessionKey. + * Supports both :thread: (most channels) and :topic: (Telegram). + */ +export function extractDeliveryInfo(sessionKey: string | undefined): { + deliveryContext: { channel?: string; to?: string; accountId?: string } | undefined; + threadId: string | undefined; +} { + if (!sessionKey) { + return { deliveryContext: undefined, threadId: undefined }; + } + const topicIndex = sessionKey.lastIndexOf(":topic:"); + const threadIndex = sessionKey.lastIndexOf(":thread:"); + const markerIndex = Math.max(topicIndex, threadIndex); + const marker = topicIndex > threadIndex ? ":topic:" : ":thread:"; + + const baseSessionKey = markerIndex === -1 ? sessionKey : sessionKey.slice(0, markerIndex); + const threadIdRaw = + markerIndex === -1 ? undefined : sessionKey.slice(markerIndex + marker.length); + const threadId = threadIdRaw?.trim() || undefined; + + let deliveryContext: { channel?: string; to?: string; accountId?: string } | undefined; + try { + const cfg = loadConfig(); + const storePath = resolveStorePath(cfg.session?.store); + const store = loadSessionStore(storePath); + let entry = store[sessionKey]; + if (!entry?.deliveryContext && markerIndex !== -1 && baseSessionKey) { + entry = store[baseSessionKey]; + } + if (entry?.deliveryContext) { + deliveryContext = { + channel: entry.deliveryContext.channel, + to: entry.deliveryContext.to, + accountId: entry.deliveryContext.accountId, + }; + } + } catch { + // ignore: best-effort + } + return { deliveryContext, threadId }; +} diff --git a/src/config/sessions/store.lock.test.ts b/src/config/sessions/store.lock.test.ts index d0d447cfcfc..83b92529532 100644 --- a/src/config/sessions/store.lock.test.ts +++ b/src/config/sessions/store.lock.test.ts @@ -50,9 +50,8 @@ describe("session store lock (Promise chain mutex)", () => { Array.from({ length: N }, (_, i) => updateSessionStore(storePath, async (store) => { const entry = store[key] as Record; - // Simulate async work so that without proper serialization - // multiple readers would see the same stale value. - await sleep(Math.random() * 3); + // Keep an async boundary so stale-read races would surface without serialization. + await Promise.resolve(); entry.counter = (entry.counter as number) + 1; entry.tag = `writer-${i}`; }), @@ -74,7 +73,7 @@ describe("session store lock (Promise chain mutex)", () => { storePath, sessionKey: key, update: async () => { - await sleep(9); + await Promise.resolve(); return { modelOverride: "model-a" }; }, }), @@ -82,7 +81,7 @@ describe("session store lock (Promise chain mutex)", () => { storePath, sessionKey: key, update: async () => { - await sleep(3); + await Promise.resolve(); return { thinkingLevel: "high" as const }; }, }), @@ -90,7 +89,7 @@ describe("session store lock (Promise chain mutex)", () => { storePath, sessionKey: key, update: async () => { - await sleep(6); + await Promise.resolve(); return { systemPromptOverride: "custom" }; }, }), @@ -165,17 +164,30 @@ describe("session store lock (Promise chain mutex)", () => { }); const order: string[] = []; + let started = 0; + let releaseBoth: (() => void) | undefined; + const gate = new Promise((resolve) => { + releaseBoth = resolve; + }); + const markStarted = () => { + started += 1; + if (started === 2) { + releaseBoth?.(); + } + }; const opA = updateSessionStore(pathA, async (store) => { order.push("a-start"); - await sleep(12); + markStarted(); + await gate; store.a = { ...store.a, modelOverride: "done-a" } as unknown as SessionEntry; order.push("a-end"); }); const opB = updateSessionStore(pathB, async (store) => { order.push("b-start"); - await sleep(3); + markStarted(); + await gate; store.b = { ...store.b, modelOverride: "done-b" } as unknown as SessionEntry; order.push("b-end"); }); @@ -211,7 +223,7 @@ describe("session store lock (Promise chain mutex)", () => { }); // Allow microtask (finally) to run. - await sleep(0); + await Promise.resolve(); expect(getSessionStoreLockQueueSizeForTest()).toBe(0); }); @@ -223,7 +235,7 @@ describe("session store lock (Promise chain mutex)", () => { throw new Error("fail"); }).catch(() => undefined); - await sleep(0); + await Promise.resolve(); expect(getSessionStoreLockQueueSizeForTest()).toBe(0); }); @@ -266,21 +278,21 @@ describe("session store lock (Promise chain mutex)", () => { const lockHolder = withSessionStoreLockForTest( storePath, async () => { - await sleep(40); + await sleep(15); }, - { timeoutMs: 2_000 }, + { timeoutMs: 1_000 }, ); const timedOut = withSessionStoreLockForTest( storePath, async () => { timedOutRan = true; }, - { timeoutMs: 20 }, + { timeoutMs: 5 }, ); await expect(timedOut).rejects.toThrow("timeout waiting for session store lock"); await lockHolder; - await sleep(8); + await sleep(2); expect(timedOutRan).toBe(false); }); @@ -291,7 +303,7 @@ describe("session store lock (Promise chain mutex)", () => { }); const write = updateSessionStore(storePath, async (store) => { - await sleep(18); + await sleep(8); store[key] = { ...store[key], modelOverride: "v" } as unknown as SessionEntry; }); @@ -303,7 +315,7 @@ describe("session store lock (Promise chain mutex)", () => { lockSeen = true; break; } catch { - await sleep(2); + await sleep(1); } } expect(lockSeen).toBe(true); diff --git a/src/config/types.models.ts b/src/config/types.models.ts index 11b6c64cbc4..ebc81f54bdd 100644 --- a/src/config/types.models.ts +++ b/src/config/types.models.ts @@ -4,13 +4,21 @@ export type ModelApi = | "anthropic-messages" | "google-generative-ai" | "github-copilot" - | "bedrock-converse-stream"; + | "bedrock-converse-stream" + | "ollama"; export type ModelCompatConfig = { supportsStore?: boolean; supportsDeveloperRole?: boolean; supportsReasoningEffort?: boolean; + supportsUsageInStreaming?: boolean; + supportsStrictMode?: boolean; maxTokensField?: "max_completion_tokens" | "max_tokens"; + thinkingFormat?: "openai" | "zai" | "qwen"; + requiresToolResultName?: boolean; + requiresAssistantAfterToolResult?: boolean; + requiresThinkingAsText?: boolean; + requiresMistralToolIds?: boolean; }; export type ModelProviderAuthMode = "api-key" | "aws-sdk" | "oauth" | "token"; diff --git a/src/config/zod-schema.core.ts b/src/config/zod-schema.core.ts index b7da9208a7a..b0562fee40c 100644 --- a/src/config/zod-schema.core.ts +++ b/src/config/zod-schema.core.ts @@ -9,6 +9,7 @@ export const ModelApiSchema = z.union([ z.literal("google-generative-ai"), z.literal("github-copilot"), z.literal("bedrock-converse-stream"), + z.literal("ollama"), ]); export const ModelCompatSchema = z @@ -16,9 +17,16 @@ export const ModelCompatSchema = z supportsStore: z.boolean().optional(), supportsDeveloperRole: z.boolean().optional(), supportsReasoningEffort: z.boolean().optional(), + supportsUsageInStreaming: z.boolean().optional(), + supportsStrictMode: z.boolean().optional(), maxTokensField: z .union([z.literal("max_completion_tokens"), z.literal("max_tokens")]) .optional(), + thinkingFormat: z.union([z.literal("openai"), z.literal("zai"), z.literal("qwen")]).optional(), + requiresToolResultName: z.boolean().optional(), + requiresAssistantAfterToolResult: z.boolean().optional(), + requiresThinkingAsText: z.boolean().optional(), + requiresMistralToolIds: z.boolean().optional(), }) .strict() .optional(); diff --git a/src/cron/isolated-agent.delivers-response-has-heartbeat-ok-but-includes.test.ts b/src/cron/isolated-agent.delivers-response-has-heartbeat-ok-but-includes.test.ts index 674763f8e79..b7e2b3c6dd8 100644 --- a/src/cron/isolated-agent.delivers-response-has-heartbeat-ok-but-includes.test.ts +++ b/src/cron/isolated-agent.delivers-response-has-heartbeat-ok-but-includes.test.ts @@ -1,10 +1,10 @@ import fs from "node:fs/promises"; +import os from "node:os"; import path from "node:path"; -import { beforeEach, describe, expect, it, vi } from "vitest"; +import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; import type { CliDeps } from "../cli/deps.js"; import type { OpenClawConfig } from "../config/config.js"; import type { CronJob } from "./types.js"; -import { withTempHome as withTempHomeBase } from "../../test/helpers/temp-home.js"; import { telegramOutbound } from "../channels/plugins/outbound/telegram.js"; import { setActivePluginRegistry } from "../plugins/runtime.js"; import { createOutboundTestPlugin, createTestRegistry } from "../test-utils/channel-plugins.js"; @@ -26,8 +26,13 @@ import { runEmbeddedPiAgent } from "../agents/pi-embedded.js"; import { runSubagentAnnounceFlow } from "../agents/subagent-announce.js"; import { runCronIsolatedAgentTurn } from "./isolated-agent.js"; +let fixtureRoot = ""; +let fixtureCount = 0; + async function withTempHome(fn: (home: string) => Promise): Promise { - return withTempHomeBase(fn, { prefix: "openclaw-cron-" }); + const home = path.join(fixtureRoot, `home-${fixtureCount++}`); + await fs.mkdir(path.join(home, ".openclaw", "agents", "main", "sessions"), { recursive: true }); + return await fn(home); } async function writeSessionStore(home: string) { @@ -87,6 +92,14 @@ function makeJob(payload: CronJob["payload"]): CronJob { } describe("runCronIsolatedAgentTurn", () => { + beforeAll(async () => { + fixtureRoot = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-cron-fixtures-")); + }); + + afterAll(async () => { + await fs.rm(fixtureRoot, { recursive: true, force: true }); + }); + beforeEach(() => { vi.mocked(runEmbeddedPiAgent).mockReset(); vi.mocked(loadModelCatalog).mockResolvedValue([]); @@ -102,7 +115,7 @@ describe("runCronIsolatedAgentTurn", () => { ); }); - it("delivers when response has HEARTBEAT_OK but includes media", async () => { + it("handles media heartbeat delivery and announce cleanup modes", async () => { await withTempHome(async (home) => { const storePath = await writeSessionStore(home); const deps: CliDeps = { @@ -115,6 +128,7 @@ describe("runCronIsolatedAgentTurn", () => { sendMessageSignal: vi.fn(), sendMessageIMessage: vi.fn(), }; + // Media should still be delivered even if text is just HEARTBEAT_OK. vi.mocked(runEmbeddedPiAgent).mockResolvedValue({ payloads: [{ text: "HEARTBEAT_OK", mediaUrl: "https://example.com/img.png" }], @@ -124,7 +138,7 @@ describe("runCronIsolatedAgentTurn", () => { }, }); - const res = await runCronIsolatedAgentTurn({ + const mediaRes = await runCronIsolatedAgentTurn({ cfg: makeCfg(home, storePath), deps, job: { @@ -139,25 +153,12 @@ describe("runCronIsolatedAgentTurn", () => { lane: "cron", }); - expect(res.status).toBe("ok"); + expect(mediaRes.status).toBe("ok"); expect(deps.sendMessageTelegram).toHaveBeenCalled(); expect(runSubagentAnnounceFlow).not.toHaveBeenCalled(); - }); - }); - it("uses shared announce flow when heartbeat ack padding exceeds configured limit", async () => { - await withTempHome(async (home) => { - const storePath = await writeSessionStore(home); - const deps: CliDeps = { - sendMessageWhatsApp: vi.fn(), - sendMessageTelegram: vi.fn().mockResolvedValue({ - messageId: "t1", - chatId: "123", - }), - sendMessageDiscord: vi.fn(), - sendMessageSignal: vi.fn(), - sendMessageIMessage: vi.fn(), - }; + vi.mocked(runSubagentAnnounceFlow).mockClear(); + vi.mocked(deps.sendMessageTelegram).mockClear(); vi.mocked(runEmbeddedPiAgent).mockResolvedValue({ payloads: [{ text: "HEARTBEAT_OK 🦞" }], meta: { @@ -175,7 +176,7 @@ describe("runCronIsolatedAgentTurn", () => { }, }; - const res = await runCronIsolatedAgentTurn({ + const keepRes = await runCronIsolatedAgentTurn({ cfg, deps, job: { @@ -190,8 +191,38 @@ describe("runCronIsolatedAgentTurn", () => { lane: "cron", }); - expect(res.status).toBe("ok"); + expect(keepRes.status).toBe("ok"); expect(runSubagentAnnounceFlow).toHaveBeenCalledTimes(1); + const keepArgs = vi.mocked(runSubagentAnnounceFlow).mock.calls[0]?.[0] as + | { cleanup?: "keep" | "delete" } + | undefined; + expect(keepArgs?.cleanup).toBe("keep"); + expect(deps.sendMessageTelegram).not.toHaveBeenCalled(); + + vi.mocked(runSubagentAnnounceFlow).mockClear(); + + const deleteRes = await runCronIsolatedAgentTurn({ + cfg, + deps, + job: { + ...makeJob({ + kind: "agentTurn", + message: "do it", + }), + deleteAfterRun: true, + delivery: { mode: "announce", channel: "telegram", to: "123" }, + }, + message: "do it", + sessionKey: "cron:job-1", + lane: "cron", + }); + + expect(deleteRes.status).toBe("ok"); + expect(runSubagentAnnounceFlow).toHaveBeenCalledTimes(1); + const deleteArgs = vi.mocked(runSubagentAnnounceFlow).mock.calls[0]?.[0] as + | { cleanup?: "keep" | "delete" } + | undefined; + expect(deleteArgs?.cleanup).toBe("delete"); expect(deps.sendMessageTelegram).not.toHaveBeenCalled(); }); }); diff --git a/src/cron/isolated-agent.skips-delivery-without-whatsapp-recipient-besteffortdeliver-true.e2e.test.ts b/src/cron/isolated-agent.skips-delivery-without-whatsapp-recipient-besteffortdeliver-true.e2e.test.ts index 4b0d04d1860..94bfd4f27bd 100644 --- a/src/cron/isolated-agent.skips-delivery-without-whatsapp-recipient-besteffortdeliver-true.e2e.test.ts +++ b/src/cron/isolated-agent.skips-delivery-without-whatsapp-recipient-besteffortdeliver-true.e2e.test.ts @@ -135,6 +135,7 @@ describe("runCronIsolatedAgentTurn", () => { }); expect(res.status).toBe("ok"); + expect(res.delivered).toBe(true); expect(runSubagentAnnounceFlow).toHaveBeenCalledTimes(1); const announceArgs = vi.mocked(runSubagentAnnounceFlow).mock.calls[0]?.[0] as | { announceType?: string } @@ -280,11 +281,56 @@ describe("runCronIsolatedAgentTurn", () => { }); expect(res.status).toBe("ok"); + expect(res.delivered).toBe(true); expect(runSubagentAnnounceFlow).not.toHaveBeenCalled(); expect(deps.sendMessageTelegram).not.toHaveBeenCalled(); }); }); + it("reports not-delivered when best-effort structured outbound sends all fail", async () => { + await withTempHome(async (home) => { + const storePath = await writeSessionStore(home); + const deps: CliDeps = { + sendMessageWhatsApp: vi.fn(), + sendMessageTelegram: vi.fn().mockRejectedValue(new Error("boom")), + sendMessageDiscord: vi.fn(), + sendMessageSignal: vi.fn(), + sendMessageIMessage: vi.fn(), + }; + vi.mocked(runEmbeddedPiAgent).mockResolvedValue({ + payloads: [{ text: "caption", mediaUrl: "https://example.com/img.png" }], + meta: { + durationMs: 5, + agentMeta: { sessionId: "s", provider: "p", model: "m" }, + }, + }); + + const res = await runCronIsolatedAgentTurn({ + cfg: makeCfg(home, storePath, { + channels: { telegram: { botToken: "t-1" } }, + }), + deps, + job: { + ...makeJob({ kind: "agentTurn", message: "do it" }), + delivery: { + mode: "announce", + channel: "telegram", + to: "123", + bestEffort: true, + }, + }, + message: "do it", + sessionKey: "cron:job-1", + lane: "cron", + }); + + expect(res.status).toBe("ok"); + expect(res.delivered).toBe(false); + expect(runSubagentAnnounceFlow).not.toHaveBeenCalled(); + expect(deps.sendMessageTelegram).toHaveBeenCalledTimes(1); + }); + }); + it("skips announce for heartbeat-only output", async () => { await withTempHome(async (home) => { const storePath = await writeSessionStore(home); diff --git a/src/cron/isolated-agent/run.ts b/src/cron/isolated-agent/run.ts index a329ef0e88e..aa97828f2b1 100644 --- a/src/cron/isolated-agent/run.ts +++ b/src/cron/isolated-agent/run.ts @@ -101,6 +101,14 @@ export type RunCronAgentTurnResult = { error?: string; sessionId?: string; sessionKey?: string; + /** + * `true` when the isolated run already delivered its output to the target + * channel (via outbound payloads, the subagent announce flow, or a matching + * messaging-tool send). Callers should skip posting a summary to the main + * session to avoid duplicate + * messages. See: https://github.com/openclaw/openclaw/issues/15692 + */ + delivered?: boolean; }; export async function runCronIsolatedAgentTurn(params: { @@ -518,6 +526,9 @@ export async function runCronIsolatedAgentTurn(params: { }), ); + // `true` means we confirmed at least one outbound send reached the target. + // Keep this strict so timer fallback can safely decide whether to wake main. + let delivered = skipMessagingToolDelivery; if (deliveryRequested && !skipHeartbeatDelivery && !skipMessagingToolDelivery) { if (resolvedDelivery.error) { if (!deliveryBestEffort) { @@ -548,7 +559,7 @@ export async function runCronIsolatedAgentTurn(params: { // for media/channel payloads so structured content is preserved. if (deliveryPayloadHasStructuredContent) { try { - await deliverOutboundPayloads({ + const deliveryResults = await deliverOutboundPayloads({ cfg: cfgWithAgentDefaults, channel: resolvedDelivery.channel, to: resolvedDelivery.to, @@ -558,6 +569,7 @@ export async function runCronIsolatedAgentTurn(params: { bestEffort: deliveryBestEffort, deps: createOutboundSendDeps(params.deps), }); + delivered = deliveryResults.length > 0; } catch (err) { if (!deliveryBestEffort) { return withRunSession({ status: "error", summary, outputText, error: String(err) }); @@ -586,7 +598,7 @@ export async function runCronIsolatedAgentTurn(params: { requesterDisplayKey: announceSessionKey, task: taskLabel, timeoutMs, - cleanup: "keep", + cleanup: params.job.deleteAfterRun ? "delete" : "keep", roundOneReply: synthesizedText, waitForCompletion: false, startedAt: runStartedAt, @@ -594,7 +606,9 @@ export async function runCronIsolatedAgentTurn(params: { outcome: { status: "ok" }, announceType: "cron job", }); - if (!didAnnounce) { + if (didAnnounce) { + delivered = true; + } else { const message = "cron announce delivery failed"; if (!deliveryBestEffort) { return withRunSession({ @@ -615,5 +629,5 @@ export async function runCronIsolatedAgentTurn(params: { } } - return withRunSession({ status: "ok", summary, outputText }); + return withRunSession({ status: "ok", summary, outputText, delivered }); } diff --git a/src/cron/service.delivery-plan.test.ts b/src/cron/service.delivery-plan.test.ts index 707868cba68..15dbc873537 100644 --- a/src/cron/service.delivery-plan.test.ts +++ b/src/cron/service.delivery-plan.test.ts @@ -89,4 +89,47 @@ describe("CronService delivery plan consistency", () => { cron.stop(); await store.cleanup(); }); + + it("does not enqueue duplicate relay when isolated run marks delivery handled", async () => { + const store = await makeStorePath(); + const enqueueSystemEvent = vi.fn(); + const requestHeartbeatNow = vi.fn(); + const runIsolatedAgentJob = vi.fn(async () => ({ + status: "ok" as const, + summary: "done", + delivered: true, + })); + const cron = new CronService({ + cronEnabled: true, + storePath: store.storePath, + log: noopLogger, + enqueueSystemEvent, + requestHeartbeatNow, + runIsolatedAgentJob, + }); + await cron.start(); + const job = await cron.add({ + name: "announce-delivered", + schedule: { kind: "every", everyMs: 60_000, anchorMs: Date.now() }, + sessionTarget: "isolated", + wakeMode: "now", + payload: { + kind: "agentTurn", + message: "hello", + }, + delivery: { channel: "telegram", to: "123" } as unknown as { + mode: "none" | "announce"; + channel?: string; + to?: string; + }, + }); + + const result = await cron.run(job.id, "force"); + expect(result).toEqual({ ok: true, ran: true }); + expect(enqueueSystemEvent).not.toHaveBeenCalled(); + expect(requestHeartbeatNow).not.toHaveBeenCalled(); + + cron.stop(); + await store.cleanup(); + }); }); diff --git a/src/cron/service.issue-regressions.test.ts b/src/cron/service.issue-regressions.test.ts index 2889d18f267..2c369b2f923 100644 --- a/src/cron/service.issue-regressions.test.ts +++ b/src/cron/service.issue-regressions.test.ts @@ -2,7 +2,7 @@ import fs from "node:fs/promises"; import os from "node:os"; import path from "node:path"; import { setTimeout as delay } from "node:timers/promises"; -import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; import type { CronJob } from "./types.js"; import { CronService } from "./service.js"; import { createCronServiceState, type CronEvent } from "./service/state.js"; @@ -16,8 +16,12 @@ const noopLogger = { trace: vi.fn(), }; +let fixtureRoot = ""; +let fixtureCount = 0; + async function makeStorePath() { - const dir = await fs.mkdtemp(path.join(os.tmpdir(), "cron-issues-")); + const dir = path.join(fixtureRoot, `case-${fixtureCount++}`); + await fs.mkdir(dir, { recursive: true }); const storePath = path.join(dir, "jobs.json"); return { storePath, @@ -50,23 +54,32 @@ function createDueIsolatedJob(params: { } describe("Cron issue regressions", () => { + beforeAll(async () => { + fixtureRoot = await fs.mkdtemp(path.join(os.tmpdir(), "cron-issues-")); + }); + beforeEach(() => { vi.useFakeTimers(); vi.setSystemTime(new Date("2026-02-06T10:05:00.000Z")); }); + afterAll(async () => { + await fs.rm(fixtureRoot, { recursive: true, force: true }); + }); + afterEach(() => { vi.useRealTimers(); vi.clearAllMocks(); }); - it("recalculates nextRunAtMs when schedule changes", async () => { + it("covers schedule updates, force runs, isolated wake scheduling, and payload patching", async () => { const store = await makeStorePath(); + const enqueueSystemEvent = vi.fn(); const cron = new CronService({ cronEnabled: true, storePath: store.storePath, log: noopLogger, - enqueueSystemEvent: vi.fn(), + enqueueSystemEvent, requestHeartbeatNow: vi.fn(), runIsolatedAgentJob: vi.fn().mockResolvedValue({ status: "ok", summary: "ok" }), }); @@ -86,51 +99,18 @@ describe("Cron issue regressions", () => { expect(updated.state.nextRunAtMs).toBe(Date.parse("2026-02-06T12:00:00.000Z")); - cron.stop(); - await store.cleanup(); - }); - - it("runs immediately with force mode even when not due", async () => { - const store = await makeStorePath(); - const enqueueSystemEvent = vi.fn(); - const cron = new CronService({ - cronEnabled: true, - storePath: store.storePath, - log: noopLogger, - enqueueSystemEvent, - requestHeartbeatNow: vi.fn(), - runIsolatedAgentJob: vi.fn().mockResolvedValue({ status: "ok", summary: "ok" }), - }); - await cron.start(); - - const created = await cron.add({ + const forceNow = await cron.add({ name: "force-now", schedule: { kind: "every", everyMs: 60_000, anchorMs: Date.now() }, sessionTarget: "main", payload: { kind: "systemEvent", text: "force" }, }); - const result = await cron.run(created.id, "force"); + const result = await cron.run(forceNow.id, "force"); expect(result).toEqual({ ok: true, ran: true }); expect(enqueueSystemEvent).toHaveBeenCalledWith("force", { agentId: undefined }); - cron.stop(); - await store.cleanup(); - }); - - it("schedules isolated jobs with next wake time", async () => { - const store = await makeStorePath(); - const cron = new CronService({ - cronEnabled: true, - storePath: store.storePath, - log: noopLogger, - enqueueSystemEvent: vi.fn(), - requestHeartbeatNow: vi.fn(), - runIsolatedAgentJob: vi.fn().mockResolvedValue({ status: "ok", summary: "ok" }), - }); - await cron.start(); - const job = await cron.add({ name: "isolated", schedule: { kind: "every", everyMs: 60_000, anchorMs: Date.now() }, @@ -142,37 +122,21 @@ describe("Cron issue regressions", () => { expect(typeof job.state.nextRunAtMs).toBe("number"); expect(typeof status.nextWakeAtMs).toBe("number"); - cron.stop(); - await store.cleanup(); - }); - - it("persists allowUnsafeExternalContent on agentTurn payload patches", async () => { - const store = await makeStorePath(); - const cron = new CronService({ - cronEnabled: true, - storePath: store.storePath, - log: noopLogger, - enqueueSystemEvent: vi.fn(), - requestHeartbeatNow: vi.fn(), - runIsolatedAgentJob: vi.fn().mockResolvedValue({ status: "ok", summary: "ok" }), - }); - await cron.start(); - - const created = await cron.add({ + const unsafeToggle = await cron.add({ name: "unsafe toggle", schedule: { kind: "every", everyMs: 60_000, anchorMs: Date.now() }, sessionTarget: "isolated", payload: { kind: "agentTurn", message: "hi" }, }); - const updated = await cron.update(created.id, { + const patched = await cron.update(unsafeToggle.id, { payload: { kind: "agentTurn", allowUnsafeExternalContent: true }, }); - expect(updated.payload.kind).toBe("agentTurn"); - if (updated.payload.kind === "agentTurn") { - expect(updated.payload.allowUnsafeExternalContent).toBe(true); - expect(updated.payload.message).toBe("hi"); + expect(patched.payload.kind).toBe("agentTurn"); + if (patched.payload.kind === "agentTurn") { + expect(patched.payload.allowUnsafeExternalContent).toBe(true); + expect(patched.payload.message).toBe("hi"); } cron.stop(); @@ -304,14 +268,10 @@ describe("Cron issue regressions", () => { await store.cleanup(); }); - it("#13845: one-shot job with lastStatus=skipped does not re-fire on restart", async () => { + it("#13845: one-shot jobs with terminal statuses do not re-fire on restart", async () => { const store = await makeStorePath(); const pastAt = Date.parse("2026-02-06T09:00:00.000Z"); - // Simulate a one-shot job that was previously skipped (e.g. main session busy). - // On the old code, runMissedJobs only checked lastStatus === "ok", so a - // skipped job would pass through and fire again on every restart. - const skippedJob: CronJob = { - id: "oneshot-skipped", + const baseJob = { name: "reminder", enabled: true, deleteAfterRun: true, @@ -321,79 +281,46 @@ describe("Cron issue regressions", () => { sessionTarget: "main", wakeMode: "now", payload: { kind: "systemEvent", text: "⏰ Reminder" }, - state: { - nextRunAtMs: pastAt, - lastStatus: "skipped", - lastRunAtMs: pastAt, - }, - }; - await fs.writeFile( - store.storePath, - JSON.stringify({ version: 1, jobs: [skippedJob] }, null, 2), - "utf-8", - ); + } as const; + for (const [id, state] of [ + [ + "oneshot-skipped", + { + nextRunAtMs: pastAt, + lastStatus: "skipped" as const, + lastRunAtMs: pastAt, + }, + ], + [ + "oneshot-errored", + { + nextRunAtMs: pastAt, + lastStatus: "error" as const, + lastRunAtMs: pastAt, + lastError: "heartbeat failed", + }, + ], + ]) { + const job: CronJob = { id, ...baseJob, state }; + await fs.writeFile( + store.storePath, + JSON.stringify({ version: 1, jobs: [job] }, null, 2), + "utf-8", + ); + const enqueueSystemEvent = vi.fn(); + const cron = new CronService({ + cronEnabled: true, + storePath: store.storePath, + log: noopLogger, + enqueueSystemEvent, + requestHeartbeatNow: vi.fn(), + runIsolatedAgentJob: vi.fn().mockResolvedValue({ status: "ok" }), + }); - const enqueueSystemEvent = vi.fn(); - const cron = new CronService({ - cronEnabled: true, - storePath: store.storePath, - log: noopLogger, - enqueueSystemEvent, - requestHeartbeatNow: vi.fn(), - runIsolatedAgentJob: vi.fn().mockResolvedValue({ status: "ok" }), - }); - - // start() calls runMissedJobs internally - await cron.start(); - - // The skipped one-shot job must NOT be re-enqueued - expect(enqueueSystemEvent).not.toHaveBeenCalled(); - - cron.stop(); - await store.cleanup(); - }); - - it("#13845: one-shot job with lastStatus=error does not re-fire on restart", async () => { - const store = await makeStorePath(); - const pastAt = Date.parse("2026-02-06T09:00:00.000Z"); - const errorJob: CronJob = { - id: "oneshot-errored", - name: "reminder", - enabled: true, - deleteAfterRun: true, - createdAtMs: pastAt - 60_000, - updatedAtMs: pastAt, - schedule: { kind: "at", at: new Date(pastAt).toISOString() }, - sessionTarget: "main", - wakeMode: "now", - payload: { kind: "systemEvent", text: "⏰ Reminder" }, - state: { - nextRunAtMs: pastAt, - lastStatus: "error", - lastRunAtMs: pastAt, - lastError: "heartbeat failed", - }, - }; - await fs.writeFile( - store.storePath, - JSON.stringify({ version: 1, jobs: [errorJob] }, null, 2), - "utf-8", - ); - - const enqueueSystemEvent = vi.fn(); - const cron = new CronService({ - cronEnabled: true, - storePath: store.storePath, - log: noopLogger, - enqueueSystemEvent, - requestHeartbeatNow: vi.fn(), - runIsolatedAgentJob: vi.fn().mockResolvedValue({ status: "ok" }), - }); - - await cron.start(); - expect(enqueueSystemEvent).not.toHaveBeenCalled(); - - cron.stop(); + await cron.start(); + expect(enqueueSystemEvent).not.toHaveBeenCalled(); + cron.stop(); + } await store.cleanup(); }); diff --git a/src/cron/service.runs-one-shot-main-job-disables-it.test.ts b/src/cron/service.runs-one-shot-main-job-disables-it.test.ts index bbee9cf7e8a..1a7c7338166 100644 --- a/src/cron/service.runs-one-shot-main-job-disables-it.test.ts +++ b/src/cron/service.runs-one-shot-main-job-disables-it.test.ts @@ -329,6 +329,48 @@ describe("CronService", () => { await store.cleanup(); }); + it("does not post isolated summary to main when run already delivered output", async () => { + const store = await makeStorePath(); + const enqueueSystemEvent = vi.fn(); + const requestHeartbeatNow = vi.fn(); + const runIsolatedAgentJob = vi.fn(async () => ({ + status: "ok" as const, + summary: "done", + delivered: true, + })); + + const cron = new CronService({ + storePath: store.storePath, + cronEnabled: true, + log: noopLogger, + enqueueSystemEvent, + requestHeartbeatNow, + runIsolatedAgentJob, + }); + + await cron.start(); + const atMs = Date.parse("2025-12-13T00:00:01.000Z"); + await cron.add({ + enabled: true, + name: "weekly delivered", + schedule: { kind: "at", at: new Date(atMs).toISOString() }, + sessionTarget: "isolated", + wakeMode: "now", + payload: { kind: "agentTurn", message: "do it" }, + delivery: { mode: "announce" }, + }); + + vi.setSystemTime(new Date("2025-12-13T00:00:01.000Z")); + await vi.runOnlyPendingTimersAsync(); + + await waitForJobs(cron, (items) => items.some((item) => item.state.lastStatus === "ok")); + expect(runIsolatedAgentJob).toHaveBeenCalledTimes(1); + expect(enqueueSystemEvent).not.toHaveBeenCalled(); + expect(requestHeartbeatNow).not.toHaveBeenCalled(); + cron.stop(); + await store.cleanup(); + }); + it("migrates legacy payload.provider to payload.channel on load", async () => { const store = await makeStorePath(); const enqueueSystemEvent = vi.fn(); diff --git a/src/cron/service/state.ts b/src/cron/service/state.ts index 025da7b3fa4..4dc1fffdf0a 100644 --- a/src/cron/service/state.ts +++ b/src/cron/service/state.ts @@ -46,6 +46,12 @@ export type CronServiceDeps = { error?: string; sessionId?: string; sessionKey?: string; + /** + * `true` when the isolated run already delivered its output to the target + * channel (including matching messaging-tool sends). See: + * https://github.com/openclaw/openclaw/issues/15692 + */ + delivered?: boolean; }>; onEvent?: (evt: CronEvent) => void; }; diff --git a/src/cron/service/timer.ts b/src/cron/service/timer.ts index 0259dfc61db..913165dcbba 100644 --- a/src/cron/service/timer.ts +++ b/src/cron/service/timer.ts @@ -483,10 +483,15 @@ async function executeJobCore( message: job.payload.message, }); - // Post a short summary back to the main session. + // Post a short summary back to the main session β€” but only when the + // isolated run did NOT already deliver its output to the target channel. + // When `res.delivered` is true the announce flow (or direct outbound + // delivery) already sent the result, so posting the summary to main + // would wake the main agent and cause a duplicate message. + // See: https://github.com/openclaw/openclaw/issues/15692 const summaryText = res.summary?.trim(); const deliveryPlan = resolveCronDeliveryPlan(job); - if (summaryText && deliveryPlan.requested) { + if (summaryText && deliveryPlan.requested && !res.delivered) { const prefix = "Cron"; const label = res.status === "error" ? `${prefix} (error): ${summaryText}` : `${prefix}: ${summaryText}`; diff --git a/src/discord/monitor/message-handler.process.test.ts b/src/discord/monitor/message-handler.process.test.ts index 619d120ca37..5e26257f317 100644 --- a/src/discord/monitor/message-handler.process.test.ts +++ b/src/discord/monitor/message-handler.process.test.ts @@ -20,7 +20,14 @@ vi.mock("../../auto-reply/reply/dispatch-from-config.js", () => ({ vi.mock("../../auto-reply/reply/reply-dispatcher.js", () => ({ createReplyDispatcherWithTyping: vi.fn(() => ({ - dispatcher: {}, + dispatcher: { + sendToolResult: vi.fn(() => true), + sendBlockReply: vi.fn(() => true), + sendFinalReply: vi.fn(() => true), + waitForIdle: vi.fn(async () => {}), + getQueuedCounts: vi.fn(() => ({ tool: 0, block: 0, final: 0 })), + markComplete: vi.fn(), + }, replyOptions: {}, markDispatchIdle: vi.fn(), })), diff --git a/src/discord/monitor/provider.ts b/src/discord/monitor/provider.ts index e61627e1555..0691e084879 100644 --- a/src/discord/monitor/provider.ts +++ b/src/discord/monitor/provider.ts @@ -4,7 +4,6 @@ import { Routes } from "discord-api-types/v10"; import { inspect } from "node:util"; import type { HistoryEntry } from "../../auto-reply/reply/history.js"; import type { OpenClawConfig, ReplyToMode } from "../../config/config.js"; -import type { RuntimeEnv } from "../../runtime.js"; import { resolveTextChunkLimit } from "../../auto-reply/chunk.js"; import { listNativeCommandSpecsForConfig } from "../../auto-reply/commands-registry.js"; import { listSkillCommandsForAgents } from "../../auto-reply/skill-commands.js"; @@ -19,6 +18,7 @@ import { danger, logVerbose, shouldLogVerbose, warn } from "../../globals.js"; import { formatErrorMessage } from "../../infra/errors.js"; import { createDiscordRetryRunner } from "../../infra/retry-policy.js"; import { createSubsystemLogger } from "../../logging/subsystem.js"; +import { createNonExitingRuntime, type RuntimeEnv } from "../../runtime.js"; import { resolveDiscordAccount } from "../accounts.js"; import { attachDiscordGatewayLogging } from "../gateway-logging.js"; import { getDiscordGatewayEmitter, waitForDiscordGatewayStop } from "../monitor.gateway.js"; @@ -137,13 +137,7 @@ export async function monitorDiscordProvider(opts: MonitorDiscordOpts = {}) { ); } - const runtime: RuntimeEnv = opts.runtime ?? { - log: console.log, - error: console.error, - exit: (code: number): never => { - throw new Error(`exit ${code}`); - }, - }; + const runtime: RuntimeEnv = opts.runtime ?? createNonExitingRuntime(); const discordCfg = account.config; const dmConfig = discordCfg.dm; diff --git a/src/docker-setup.test.ts b/src/docker-setup.test.ts index c3b9f19dd64..07b349c76d8 100644 --- a/src/docker-setup.test.ts +++ b/src/docker-setup.test.ts @@ -85,47 +85,53 @@ function resolveBashForCompatCheck(): string | null { } describe("docker-setup.sh", () => { - it("handles unset optional env vars under strict mode", async () => { + it("handles env defaults, home-volume mounts, and apt build args", async () => { const sandbox = await createDockerSetupSandbox(); - const env = createEnv(sandbox, { - OPENCLAW_DOCKER_APT_PACKAGES: undefined, - OPENCLAW_EXTRA_MOUNTS: undefined, - OPENCLAW_HOME_VOLUME: undefined, - }); - const result = spawnSync("bash", [sandbox.scriptPath], { + const defaultsResult = spawnSync("bash", [sandbox.scriptPath], { cwd: sandbox.rootDir, - env, + env: createEnv(sandbox, { + OPENCLAW_DOCKER_APT_PACKAGES: undefined, + OPENCLAW_EXTRA_MOUNTS: undefined, + OPENCLAW_HOME_VOLUME: undefined, + }), encoding: "utf8", }); + expect(defaultsResult.status).toBe(0); + const defaultsEnvFile = await readFile(join(sandbox.rootDir, ".env"), "utf8"); + expect(defaultsEnvFile).toContain("OPENCLAW_DOCKER_APT_PACKAGES="); + expect(defaultsEnvFile).toContain("OPENCLAW_EXTRA_MOUNTS="); + expect(defaultsEnvFile).toContain("OPENCLAW_HOME_VOLUME="); - expect(result.status).toBe(0); - - const envFile = await readFile(join(sandbox.rootDir, ".env"), "utf8"); - expect(envFile).toContain("OPENCLAW_DOCKER_APT_PACKAGES="); - expect(envFile).toContain("OPENCLAW_EXTRA_MOUNTS="); - expect(envFile).toContain("OPENCLAW_HOME_VOLUME="); - }); - - it("supports a home volume when extra mounts are empty", async () => { - const sandbox = await createDockerSetupSandbox(); - const env = createEnv(sandbox, { - OPENCLAW_EXTRA_MOUNTS: "", - OPENCLAW_HOME_VOLUME: "openclaw-home", - }); - - const result = spawnSync("bash", [sandbox.scriptPath], { + const homeVolumeResult = spawnSync("bash", [sandbox.scriptPath], { cwd: sandbox.rootDir, - env, + env: createEnv(sandbox, { + OPENCLAW_EXTRA_MOUNTS: "", + OPENCLAW_HOME_VOLUME: "openclaw-home", + }), encoding: "utf8", }); - - expect(result.status).toBe(0); - + expect(homeVolumeResult.status).toBe(0); const extraCompose = await readFile(join(sandbox.rootDir, "docker-compose.extra.yml"), "utf8"); expect(extraCompose).toContain("openclaw-home:/home/node"); expect(extraCompose).toContain("volumes:"); expect(extraCompose).toContain("openclaw-home:"); + + await writeFile(sandbox.logPath, ""); + const aptResult = spawnSync("bash", [sandbox.scriptPath], { + cwd: sandbox.rootDir, + env: createEnv(sandbox, { + OPENCLAW_DOCKER_APT_PACKAGES: "ffmpeg build-essential", + OPENCLAW_EXTRA_MOUNTS: "", + OPENCLAW_HOME_VOLUME: "", + }), + encoding: "utf8", + }); + expect(aptResult.status).toBe(0); + const aptEnvFile = await readFile(join(sandbox.rootDir, ".env"), "utf8"); + expect(aptEnvFile).toContain("OPENCLAW_DOCKER_APT_PACKAGES=ffmpeg build-essential"); + const log = await readFile(sandbox.logPath, "utf8"); + expect(log).toContain("--build-arg OPENCLAW_DOCKER_APT_PACKAGES=ffmpeg build-essential"); }); it("avoids associative arrays so the script remains Bash 3.2-compatible", async () => { @@ -146,42 +152,12 @@ describe("docker-setup.sh", () => { return; } - const sandbox = await createDockerSetupSandbox(); - const env = createEnv(sandbox, { - OPENCLAW_EXTRA_MOUNTS: "", - OPENCLAW_HOME_VOLUME: "", - }); - const result = spawnSync(systemBash, [sandbox.scriptPath], { - cwd: sandbox.rootDir, - env, + const syntaxCheck = spawnSync(systemBash, ["-n", join(repoRoot, "docker-setup.sh")], { encoding: "utf8", }); - expect(result.status).toBe(0); - expect(result.stderr).not.toContain("declare: -A: invalid option"); - }); - - it("plumbs OPENCLAW_DOCKER_APT_PACKAGES into .env and docker build args", async () => { - const sandbox = await createDockerSetupSandbox(); - const env = createEnv(sandbox, { - OPENCLAW_DOCKER_APT_PACKAGES: "ffmpeg build-essential", - OPENCLAW_EXTRA_MOUNTS: "", - OPENCLAW_HOME_VOLUME: "", - }); - - const result = spawnSync("bash", [sandbox.scriptPath], { - cwd: sandbox.rootDir, - env, - encoding: "utf8", - }); - - expect(result.status).toBe(0); - - const envFile = await readFile(join(sandbox.rootDir, ".env"), "utf8"); - expect(envFile).toContain("OPENCLAW_DOCKER_APT_PACKAGES=ffmpeg build-essential"); - - const log = await readFile(sandbox.logPath, "utf8"); - expect(log).toContain("--build-arg OPENCLAW_DOCKER_APT_PACKAGES=ffmpeg build-essential"); + expect(syntaxCheck.status).toBe(0); + expect(syntaxCheck.stderr).not.toContain("declare: -A: invalid option"); }); it("keeps docker-compose gateway command in sync", async () => { diff --git a/src/entry.ts b/src/entry.ts index 81ce2c3e607..46fc96d1e56 100644 --- a/src/entry.ts +++ b/src/entry.ts @@ -3,6 +3,7 @@ import { spawn } from "node:child_process"; import path from "node:path"; import process from "node:process"; import { applyCliProfileEnv, parseCliProfileArgs } from "./cli/profile.js"; +import { shouldSkipRespawnForArgv } from "./cli/respawn-policy.js"; import { isTruthyEnvValue, normalizeEnv } from "./infra/env.js"; import { installProcessWarningFilter } from "./infra/warning-filter.js"; import { attachChildProcessBridge } from "./process/child-process-bridge.js"; @@ -32,6 +33,9 @@ function hasExperimentalWarningSuppressed(): boolean { } function ensureExperimentalWarningSuppressed(): boolean { + if (shouldSkipRespawnForArgv(process.argv)) { + return false; + } if (isTruthyEnvValue(process.env.OPENCLAW_NO_RESPAWN)) { return false; } diff --git a/src/gateway/server-channels.ts b/src/gateway/server-channels.ts index 73a6a11cd7a..1ed2250547f 100644 --- a/src/gateway/server-channels.ts +++ b/src/gateway/server-channels.ts @@ -180,8 +180,12 @@ export function createChannelManager(opts: ChannelManagerOptions): ChannelManage const stopChannel = async (channelId: ChannelId, accountId?: string) => { const plugin = getChannelPlugin(channelId); - const cfg = loadConfig(); const store = getStore(channelId); + // Fast path: nothing running and no explicit plugin shutdown hook to run. + if (!plugin?.gateway?.stopAccount && store.aborts.size === 0 && store.tasks.size === 0) { + return; + } + const cfg = loadConfig(); const knownIds = new Set([ ...store.aborts.keys(), ...store.tasks.keys(), diff --git a/src/gateway/server-http.hooks-request-timeout.test.ts b/src/gateway/server-http.hooks-request-timeout.test.ts new file mode 100644 index 00000000000..e76c243d5c1 --- /dev/null +++ b/src/gateway/server-http.hooks-request-timeout.test.ts @@ -0,0 +1,99 @@ +import type { IncomingMessage, ServerResponse } from "node:http"; +import { beforeEach, describe, expect, test, vi } from "vitest"; +import type { createSubsystemLogger } from "../logging/subsystem.js"; +import type { HooksConfigResolved } from "./hooks.js"; + +const { readJsonBodyMock } = vi.hoisted(() => ({ + readJsonBodyMock: vi.fn(), +})); + +vi.mock("./hooks.js", async (importOriginal) => { + const actual = await importOriginal(); + return { + ...actual, + readJsonBody: readJsonBodyMock, + }; +}); + +import { createHooksRequestHandler } from "./server-http.js"; + +function createHooksConfig(): HooksConfigResolved { + return { + basePath: "/hooks", + token: "hook-secret", + maxBodyBytes: 1024, + mappings: [], + agentPolicy: { + defaultAgentId: "main", + knownAgentIds: new Set(["main"]), + allowedAgentIds: undefined, + }, + sessionPolicy: { + allowRequestSessionKey: false, + defaultSessionKey: undefined, + allowedSessionKeyPrefixes: undefined, + }, + }; +} + +function createRequest(): IncomingMessage { + return { + method: "POST", + url: "/hooks/wake", + headers: { + host: "127.0.0.1:18789", + authorization: "Bearer hook-secret", + }, + socket: { remoteAddress: "127.0.0.1" }, + } as IncomingMessage; +} + +function createResponse(): { + res: ServerResponse; + end: ReturnType; + setHeader: ReturnType; +} { + const setHeader = vi.fn(); + const end = vi.fn(); + const res = { + statusCode: 200, + setHeader, + end, + } as unknown as ServerResponse; + return { res, end, setHeader }; +} + +describe("createHooksRequestHandler timeout status mapping", () => { + beforeEach(() => { + readJsonBodyMock.mockReset(); + }); + + test("returns 408 for request body timeout", async () => { + readJsonBodyMock.mockResolvedValue({ ok: false, error: "request body timeout" }); + const dispatchWakeHook = vi.fn(); + const dispatchAgentHook = vi.fn(() => "run-1"); + const handler = createHooksRequestHandler({ + getHooksConfig: () => createHooksConfig(), + bindHost: "127.0.0.1", + port: 18789, + logHooks: { + warn: vi.fn(), + debug: vi.fn(), + info: vi.fn(), + error: vi.fn(), + } as unknown as ReturnType, + dispatchWakeHook, + dispatchAgentHook, + }); + const req = createRequest(); + const { res, end } = createResponse(); + + const handled = await handler(req, res); + + expect(handled).toBe(true); + expect(res.statusCode).toBe(408); + expect(end).toHaveBeenCalledWith(JSON.stringify({ ok: false, error: "request body timeout" })); + expect(dispatchWakeHook).not.toHaveBeenCalled(); + expect(dispatchAgentHook).not.toHaveBeenCalled(); + }); +}); diff --git a/src/gateway/server-http.ts b/src/gateway/server-http.ts index 7b5630d1a11..1bdd5acf240 100644 --- a/src/gateway/server-http.ts +++ b/src/gateway/server-http.ts @@ -207,13 +207,34 @@ export function createHooksRequestHandler( nowMs: number, ): { throttled: boolean; retryAfterSeconds?: number } => { if (!hookAuthFailures.has(clientKey) && hookAuthFailures.size >= HOOK_AUTH_FAILURE_TRACK_MAX) { - hookAuthFailures.clear(); + // Prune expired entries instead of clearing all state. + for (const [key, entry] of hookAuthFailures) { + if (nowMs - entry.windowStartedAtMs >= HOOK_AUTH_FAILURE_WINDOW_MS) { + hookAuthFailures.delete(key); + } + } + // If still at capacity after pruning, drop the oldest half. + if (hookAuthFailures.size >= HOOK_AUTH_FAILURE_TRACK_MAX) { + let toRemove = Math.floor(hookAuthFailures.size / 2); + for (const key of hookAuthFailures.keys()) { + if (toRemove <= 0) { + break; + } + hookAuthFailures.delete(key); + toRemove--; + } + } } const current = hookAuthFailures.get(clientKey); const expired = !current || nowMs - current.windowStartedAtMs >= HOOK_AUTH_FAILURE_WINDOW_MS; const next: HookAuthFailure = expired ? { count: 1, windowStartedAtMs: nowMs } : { count: current.count + 1, windowStartedAtMs: current.windowStartedAtMs }; + // Delete-before-set refreshes Map insertion order so recently-active + // clients are not evicted before dormant ones during oldest-half eviction. + if (hookAuthFailures.has(clientKey)) { + hookAuthFailures.delete(clientKey); + } hookAuthFailures.set(clientKey, next); if (next.count <= HOOK_AUTH_FAILURE_LIMIT) { return { throttled: false }; diff --git a/src/gateway/server-methods/config.ts b/src/gateway/server-methods/config.ts index d4be1a8667e..30ba6bc9832 100644 --- a/src/gateway/server-methods/config.ts +++ b/src/gateway/server-methods/config.ts @@ -6,6 +6,7 @@ import { loadConfig, parseConfigJson5, readConfigFileSnapshot, + readConfigFileSnapshotForWrite, resolveConfigSnapshotHash, validateConfigObjectWithPlugins, writeConfigFile, @@ -18,6 +19,7 @@ import { restoreRedactedValues, } from "../../config/redact-snapshot.js"; import { buildConfigSchema, type ConfigSchemaResponse } from "../../config/schema.js"; +import { extractDeliveryInfo } from "../../config/sessions.js"; import { formatDoctorNonInteractiveHint, type RestartSentinelPayload, @@ -169,7 +171,7 @@ export const configHandlers: GatewayRequestHandlers = { ); return; } - const snapshot = await readConfigFileSnapshot(); + const { snapshot, writeOptions } = await readConfigFileSnapshotForWrite(); if (!requireConfigBaseHash(params, snapshot, respond)) { return; } @@ -208,7 +210,7 @@ export const configHandlers: GatewayRequestHandlers = { ); return; } - await writeConfigFile(validated.config); + await writeConfigFile(validated.config, writeOptions); respond( true, { @@ -231,7 +233,7 @@ export const configHandlers: GatewayRequestHandlers = { ); return; } - const snapshot = await readConfigFileSnapshot(); + const { snapshot, writeOptions } = await readConfigFileSnapshotForWrite(); if (!requireConfigBaseHash(params, snapshot, respond)) { return; } @@ -299,7 +301,7 @@ export const configHandlers: GatewayRequestHandlers = { ); return; } - await writeConfigFile(validated.config); + await writeConfigFile(validated.config, writeOptions); const sessionKey = typeof (params as { sessionKey?: unknown }).sessionKey === "string" @@ -315,11 +317,17 @@ export const configHandlers: GatewayRequestHandlers = { ? Math.max(0, Math.floor(restartDelayMsRaw)) : undefined; + // Extract deliveryContext + threadId for routing after restart + // Supports both :thread: (most channels) and :topic: (Telegram) + const { deliveryContext, threadId } = extractDeliveryInfo(sessionKey); + const payload: RestartSentinelPayload = { - kind: "config-apply", + kind: "config-patch", status: "ok", ts: Date.now(), sessionKey, + deliveryContext, + threadId, message: note ?? null, doctorHint: formatDoctorNonInteractiveHint(), stats: { @@ -364,7 +372,7 @@ export const configHandlers: GatewayRequestHandlers = { ); return; } - const snapshot = await readConfigFileSnapshot(); + const { snapshot, writeOptions } = await readConfigFileSnapshotForWrite(); if (!requireConfigBaseHash(params, snapshot, respond)) { return; } @@ -406,7 +414,7 @@ export const configHandlers: GatewayRequestHandlers = { ); return; } - await writeConfigFile(validated.config); + await writeConfigFile(validated.config, writeOptions); const sessionKey = typeof (params as { sessionKey?: unknown }).sessionKey === "string" @@ -422,11 +430,18 @@ export const configHandlers: GatewayRequestHandlers = { ? Math.max(0, Math.floor(restartDelayMsRaw)) : undefined; + // Extract deliveryContext + threadId for routing after restart + // Supports both :thread: (most channels) and :topic: (Telegram) + const { deliveryContext: deliveryContextApply, threadId: threadIdApply } = + extractDeliveryInfo(sessionKey); + const payload: RestartSentinelPayload = { kind: "config-apply", status: "ok", ts: Date.now(), sessionKey, + deliveryContext: deliveryContextApply, + threadId: threadIdApply, message: note ?? null, doctorHint: formatDoctorNonInteractiveHint(), stats: { diff --git a/src/gateway/server-methods/skills.update.normalizes-api-key.test.ts b/src/gateway/server-methods/skills.update.normalizes-api-key.test.ts index 45b9d719e7c..ac4dc516722 100644 --- a/src/gateway/server-methods/skills.update.normalizes-api-key.test.ts +++ b/src/gateway/server-methods/skills.update.normalizes-api-key.test.ts @@ -15,10 +15,11 @@ vi.mock("../../config/config.js", () => { }; }); +const { skillsHandlers } = await import("./skills.js"); + describe("skills.update", () => { it("strips embedded CR/LF from apiKey", async () => { writtenConfig = null; - const { skillsHandlers } = await import("./skills.js"); let ok: boolean | null = null; let error: unknown = null; diff --git a/src/gateway/server-reload-handlers.ts b/src/gateway/server-reload-handlers.ts index 393a38cf778..6a2dfd2cb27 100644 --- a/src/gateway/server-reload-handlers.ts +++ b/src/gateway/server-reload-handlers.ts @@ -2,15 +2,18 @@ import type { CliDeps } from "../cli/deps.js"; import type { loadConfig } from "../config/config.js"; import type { HeartbeatRunner } from "../infra/heartbeat-runner.js"; import type { ChannelKind, GatewayReloadPlan } from "./config-reload.js"; +import { getActiveEmbeddedRunCount } from "../agents/pi-embedded-runner/runs.js"; +import { getTotalPendingReplies } from "../auto-reply/reply/dispatcher-registry.js"; import { resolveAgentMaxConcurrent, resolveSubagentMaxConcurrent } from "../config/agent-limits.js"; import { startGmailWatcher, stopGmailWatcher } from "../hooks/gmail-watcher.js"; import { isTruthyEnvValue } from "../infra/env.js"; import { resetDirectoryCache } from "../infra/outbound/target-resolver.js"; import { - authorizeGatewaySigusr1Restart, + deferGatewayRestartUntilIdle, + emitGatewayRestart, setGatewaySigusr1RestartPolicy, } from "../infra/restart.js"; -import { setCommandLaneConcurrency } from "../process/command-queue.js"; +import { setCommandLaneConcurrency, getTotalQueueSize } from "../process/command-queue.js"; import { CommandLane } from "../process/lanes.js"; import { resolveHooksConfig } from "./hooks.js"; import { startBrowserControlServerIfEnabled } from "./server-browser.js"; @@ -140,6 +143,8 @@ export function createGatewayReloadHandlers(params: { params.setState(nextState); }; + let restartPending = false; + const requestGatewayRestart = ( plan: GatewayReloadPlan, nextConfig: ReturnType, @@ -148,13 +153,82 @@ export function createGatewayReloadHandlers(params: { const reasons = plan.restartReasons.length ? plan.restartReasons.join(", ") : plan.changedPaths.join(", "); - params.logReload.warn(`config change requires gateway restart (${reasons})`); + if (process.listenerCount("SIGUSR1") === 0) { params.logReload.warn("no SIGUSR1 listener found; restart skipped"); return; } - authorizeGatewaySigusr1Restart(); - process.emit("SIGUSR1"); + + const getActiveCounts = () => { + const queueSize = getTotalQueueSize(); + const pendingReplies = getTotalPendingReplies(); + const embeddedRuns = getActiveEmbeddedRunCount(); + return { + queueSize, + pendingReplies, + embeddedRuns, + totalActive: queueSize + pendingReplies + embeddedRuns, + }; + }; + const formatActiveDetails = (counts: ReturnType) => { + const details = []; + if (counts.queueSize > 0) { + details.push(`${counts.queueSize} operation(s)`); + } + if (counts.pendingReplies > 0) { + details.push(`${counts.pendingReplies} reply(ies)`); + } + if (counts.embeddedRuns > 0) { + details.push(`${counts.embeddedRuns} embedded run(s)`); + } + return details; + }; + const active = getActiveCounts(); + + if (active.totalActive > 0) { + // Avoid spinning up duplicate polling loops from repeated config changes. + if (restartPending) { + params.logReload.info( + `config change requires gateway restart (${reasons}) β€” already waiting for operations to complete`, + ); + return; + } + restartPending = true; + const initialDetails = formatActiveDetails(active); + params.logReload.warn( + `config change requires gateway restart (${reasons}) β€” deferring until ${initialDetails.join(", ")} complete`, + ); + + deferGatewayRestartUntilIdle({ + getPendingCount: () => getActiveCounts().totalActive, + hooks: { + onReady: () => { + restartPending = false; + params.logReload.info("all operations and replies completed; restarting gateway now"); + }, + onTimeout: (_pending, elapsedMs) => { + const remaining = formatActiveDetails(getActiveCounts()); + restartPending = false; + params.logReload.warn( + `restart timeout after ${elapsedMs}ms with ${remaining.join(", ")} still active; restarting anyway`, + ); + }, + onCheckError: (err) => { + restartPending = false; + params.logReload.warn( + `restart deferral check failed (${String(err)}); restarting gateway now`, + ); + }, + }, + }); + } else { + // No active operations or pending replies, restart immediately + params.logReload.warn(`config change requires gateway restart (${reasons})`); + const emitted = emitGatewayRestart(); + if (!emitted) { + params.logReload.info("gateway restart already scheduled; skipping duplicate signal"); + } + } }; return { applyHotReload, requestGatewayRestart }; diff --git a/src/gateway/server-reload.config-during-reply.test.ts b/src/gateway/server-reload.config-during-reply.test.ts new file mode 100644 index 00000000000..3d8be89749b --- /dev/null +++ b/src/gateway/server-reload.config-during-reply.test.ts @@ -0,0 +1,105 @@ +/** + * E2E test for config reload during active reply sending. + * Tests that gateway restart is properly deferred until replies are sent. + */ +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { + clearAllDispatchers, + getTotalPendingReplies, +} from "../auto-reply/reply/dispatcher-registry.js"; +import { createReplyDispatcher } from "../auto-reply/reply/reply-dispatcher.js"; +import { getTotalQueueSize } from "../process/command-queue.js"; + +// Helper to flush all pending microtasks +async function flushMicrotasks() { + for (let i = 0; i < 10; i++) { + await Promise.resolve(); + } +} + +describe("gateway config reload during reply", () => { + beforeEach(() => { + vi.clearAllMocks(); + }); + + afterEach(async () => { + vi.restoreAllMocks(); + // Wait for any pending microtasks (from markComplete()) to complete + await flushMicrotasks(); + clearAllDispatchers(); + }); + + it("should defer restart until reply dispatcher completes", async () => { + // Create a dispatcher (simulating message handling) + let deliveredReplies: string[] = []; + const dispatcher = createReplyDispatcher({ + deliver: async (payload) => { + // Keep delivery asynchronous without real wall-clock delay. + await Promise.resolve(); + deliveredReplies.push(payload.text ?? ""); + }, + onError: (err) => { + throw err; + }, + }); + + // Initially: pending=1 (reservation) + expect(getTotalPendingReplies()).toBe(1); + + // Simulate command finishing and enqueuing reply + dispatcher.sendFinalReply({ text: "Configuration updated successfully!" }); + + // Now: pending=2 (reservation + 1 enqueued reply) + expect(getTotalPendingReplies()).toBe(2); + + // Mark dispatcher complete (flags reservation for cleanup on last delivery) + dispatcher.markComplete(); + + // Reservation is still counted until the delivery .finally() clears it, + // but the important invariant is pending > 0 while delivery is in flight. + expect(getTotalPendingReplies()).toBeGreaterThan(0); + + // At this point, if gateway restart was requested, it should defer + // because getTotalPendingReplies() > 0 + + // Wait for reply to be delivered + await dispatcher.waitForIdle(); + + // Now: pending=0 (reply sent) + expect(getTotalPendingReplies()).toBe(0); + expect(deliveredReplies).toEqual(["Configuration updated successfully!"]); + + // Now restart can proceed safely + expect(getTotalQueueSize()).toBe(0); + expect(getTotalPendingReplies()).toBe(0); + }); + + it("should handle dispatcher reservation correctly when no replies sent", async () => { + const { createReplyDispatcher } = await import("../auto-reply/reply/reply-dispatcher.js"); + + let deliverCalled = false; + const dispatcher = createReplyDispatcher({ + deliver: async () => { + deliverCalled = true; + }, + }); + + // Initially: pending=1 (reservation) + expect(getTotalPendingReplies()).toBe(1); + + // Mark complete without sending any replies + dispatcher.markComplete(); + + // Reservation is cleared via microtask β€” flush it + await flushMicrotasks(); + + // Now: pending=0 (reservation cleared, no replies were enqueued) + expect(getTotalPendingReplies()).toBe(0); + + // Wait for idle (should resolve immediately since no replies) + await dispatcher.waitForIdle(); + + expect(deliverCalled).toBe(false); + expect(getTotalPendingReplies()).toBe(0); + }); +}); diff --git a/src/gateway/server-reload.integration.test.ts b/src/gateway/server-reload.integration.test.ts new file mode 100644 index 00000000000..9d788ad4947 --- /dev/null +++ b/src/gateway/server-reload.integration.test.ts @@ -0,0 +1,120 @@ +/** + * Integration test simulating full message handling + config change + reply flow. + * This tests the complete scenario where a user configures an adapter via chat + * and ensures they get a reply before the gateway restarts. + */ +import { describe, expect, it, vi, beforeEach, afterEach } from "vitest"; +import { + clearAllDispatchers, + getTotalPendingReplies, +} from "../auto-reply/reply/dispatcher-registry.js"; +import { createReplyDispatcher } from "../auto-reply/reply/reply-dispatcher.js"; +import { getTotalQueueSize } from "../process/command-queue.js"; + +describe("gateway restart deferral integration", () => { + beforeEach(() => { + vi.clearAllMocks(); + }); + + afterEach(async () => { + vi.restoreAllMocks(); + // Wait for any pending microtasks (from markComplete()) to complete + await Promise.resolve(); + clearAllDispatchers(); + }); + + it("should defer restart until dispatcher completes with reply", async () => { + const events: string[] = []; + + // T=0: Message received β€” dispatcher created (pending=1 reservation) + events.push("message-received"); + const deliveredReplies: Array<{ text: string; timestamp: number }> = []; + const dispatcher = createReplyDispatcher({ + deliver: async (payload) => { + // Keep delivery asynchronous without real wall-clock delay. + await Promise.resolve(); + deliveredReplies.push({ + text: payload.text ?? "", + timestamp: Date.now(), + }); + events.push(`reply-delivered: ${payload.text}`); + }, + }); + events.push("dispatcher-created"); + + // T=1: Config change detected + events.push("config-change-detected"); + + // Check if restart should be deferred + const queueSize = getTotalQueueSize(); + const pendingReplies = getTotalPendingReplies(); + const totalActive = queueSize + pendingReplies; + + events.push(`defer-check: queue=${queueSize} pending=${pendingReplies} total=${totalActive}`); + + // Should defer because dispatcher has reservation + expect(totalActive).toBeGreaterThan(0); + expect(pendingReplies).toBe(1); // reservation + + if (totalActive > 0) { + events.push("restart-deferred"); + } + + // T=2: Command finishes, enqueue replies + dispatcher.sendFinalReply({ text: "Adapter configured successfully!" }); + dispatcher.sendFinalReply({ text: "Gateway will restart to apply changes." }); + events.push("replies-enqueued"); + + // Now pending should be 3 (reservation + 2 replies) + expect(getTotalPendingReplies()).toBe(3); + + // Mark command complete (flags reservation for cleanup on last delivery) + dispatcher.markComplete(); + events.push("command-complete"); + + // Reservation still counted until delivery .finally() clears it, + // but the important invariant is pending > 0 while deliveries are in flight. + expect(getTotalPendingReplies()).toBeGreaterThan(0); + + // T=3: Wait for replies to be delivered + await dispatcher.waitForIdle(); + events.push("dispatcher-idle"); + + // Replies should be delivered + expect(deliveredReplies).toHaveLength(2); + expect(deliveredReplies[0].text).toBe("Adapter configured successfully!"); + expect(deliveredReplies[1].text).toBe("Gateway will restart to apply changes."); + + // Pending should be 0 + expect(getTotalPendingReplies()).toBe(0); + + // T=4: Check if restart can proceed + const finalQueueSize = getTotalQueueSize(); + const finalPendingReplies = getTotalPendingReplies(); + const finalTotalActive = finalQueueSize + finalPendingReplies; + + events.push( + `restart-check: queue=${finalQueueSize} pending=${finalPendingReplies} total=${finalTotalActive}`, + ); + + // Everything should be idle now + expect(finalTotalActive).toBe(0); + events.push("restart-can-proceed"); + + // Verify event sequence + expect(events).toEqual([ + "message-received", + "dispatcher-created", + "config-change-detected", + "defer-check: queue=0 pending=1 total=1", + "restart-deferred", + "replies-enqueued", + "command-complete", + "reply-delivered: Adapter configured successfully!", + "reply-delivered: Gateway will restart to apply changes.", + "dispatcher-idle", + "restart-check: queue=0 pending=0 total=0", + "restart-can-proceed", + ]); + }); +}); diff --git a/src/gateway/server-reload.real-scenario.test.ts b/src/gateway/server-reload.real-scenario.test.ts new file mode 100644 index 00000000000..f26f660dc79 --- /dev/null +++ b/src/gateway/server-reload.real-scenario.test.ts @@ -0,0 +1,136 @@ +/** + * REAL scenario test - simulates actual message handling with config changes. + * This test MUST fail if "imsg rpc not running" would occur in production. + */ +import { describe, expect, it, vi, beforeEach, afterEach } from "vitest"; +import { + clearAllDispatchers, + getTotalPendingReplies, +} from "../auto-reply/reply/dispatcher-registry.js"; +import { createReplyDispatcher } from "../auto-reply/reply/reply-dispatcher.js"; + +function createDeferred() { + let resolve!: (value: T | PromiseLike) => void; + let reject!: (reason?: unknown) => void; + const promise = new Promise((res, rej) => { + resolve = res; + reject = rej; + }); + return { promise, resolve, reject }; +} + +describe("real scenario: config change during message processing", () => { + let replyErrors: string[] = []; + + beforeEach(() => { + vi.clearAllMocks(); + replyErrors = []; + }); + + afterEach(async () => { + vi.restoreAllMocks(); + // Wait for any pending microtasks (from markComplete()) to complete + await Promise.resolve(); + clearAllDispatchers(); + }); + + it("should NOT restart gateway while reply delivery is in flight", async () => { + let rpcConnected = true; + const deliveredReplies: string[] = []; + const deliveryStarted = createDeferred(); + const allowDelivery = createDeferred(); + + // Hold delivery open so restart checks run while reply is in-flight. + const dispatcher = createReplyDispatcher({ + deliver: async (payload) => { + if (!rpcConnected) { + const error = "Error: imsg rpc not running"; + replyErrors.push(error); + throw new Error(error); + } + deliveryStarted.resolve(); + await allowDelivery.promise; + deliveredReplies.push(payload.text ?? ""); + }, + onError: () => { + // Swallow delivery errors so the test can assert on replyErrors + }, + }); + + // Enqueue reply and immediately clear the reservation. + // This is the critical sequence: after markComplete(), the ONLY thing + // keeping pending > 0 is the in-flight delivery itself. + dispatcher.sendFinalReply({ text: "Configuration updated!" }); + dispatcher.markComplete(); + await deliveryStarted.promise; + + // At this point: markComplete flagged, delivery is in flight. + // pending > 0 because the in-flight delivery keeps it alive. + const pendingDuringDelivery = getTotalPendingReplies(); + expect(pendingDuringDelivery).toBeGreaterThan(0); + + // Simulate restart checks while delivery is in progress. + // If the tracking is broken, pending would be 0 and we'd restart. + let restartTriggered = false; + for (let i = 0; i < 3; i++) { + await Promise.resolve(); + const pending = getTotalPendingReplies(); + if (pending === 0) { + restartTriggered = true; + rpcConnected = false; + break; + } + } + + allowDelivery.resolve(); + // Wait for delivery to complete + await dispatcher.waitForIdle(); + + // Now pending should be 0 β€” restart can proceed + expect(getTotalPendingReplies()).toBe(0); + + // CRITICAL: delivery must have succeeded without RPC being killed + expect(restartTriggered).toBe(false); + expect(replyErrors).toEqual([]); + expect(deliveredReplies).toEqual(["Configuration updated!"]); + }); + + it("should keep pending > 0 until reply is actually enqueued", async () => { + const allowDelivery = createDeferred(); + + const dispatcher = createReplyDispatcher({ + deliver: async (_payload) => { + await allowDelivery.promise; + }, + }); + + // Initially: pending=1 (reservation) + expect(getTotalPendingReplies()).toBe(1); + + // Simulate command processing delay BEFORE reply is enqueued + await Promise.resolve(); + + // During this delay, pending should STILL be 1 (reservation active) + expect(getTotalPendingReplies()).toBe(1); + + // Now enqueue reply + dispatcher.sendFinalReply({ text: "Reply" }); + + // Now pending should be 2 (reservation + reply) + expect(getTotalPendingReplies()).toBe(2); + + // Mark complete + dispatcher.markComplete(); + + // After markComplete, pending should still be > 0 if reply hasn't sent yet + const pendingAfterMarkComplete = getTotalPendingReplies(); + expect(pendingAfterMarkComplete).toBeGreaterThan(0); + + allowDelivery.resolve(); + // Wait for reply to send + await dispatcher.waitForIdle(); + + // Now pending should be 0 + expect(getTotalPendingReplies()).toBe(0); + }); +}); diff --git a/src/gateway/server-restart-sentinel.ts b/src/gateway/server-restart-sentinel.ts index 2600a0b6380..901465b5684 100644 --- a/src/gateway/server-restart-sentinel.ts +++ b/src/gateway/server-restart-sentinel.ts @@ -1,8 +1,8 @@ import type { CliDeps } from "../cli/deps.js"; import { resolveAnnounceTargetFromKey } from "../agents/tools/sessions-send-helpers.js"; import { normalizeChannelId } from "../channels/plugins/index.js"; -import { agentCommand } from "../commands/agent.js"; import { resolveMainSessionKeyFromConfig } from "../config/sessions.js"; +import { deliverOutboundPayloads } from "../infra/outbound/deliver.js"; import { resolveOutboundTarget } from "../infra/outbound/targets.js"; import { consumeRestartSentinel, @@ -10,11 +10,10 @@ import { summarizeRestartSentinel, } from "../infra/restart-sentinel.js"; import { enqueueSystemEvent } from "../infra/system-events.js"; -import { defaultRuntime } from "../runtime.js"; import { deliveryContextFromSession, mergeDeliveryContext } from "../utils/delivery-context.js"; import { loadSessionEntry } from "./session-utils.js"; -export async function scheduleRestartSentinelWake(params: { deps: CliDeps }) { +export async function scheduleRestartSentinelWake(_params: { deps: CliDeps }) { const sentinel = await consumeRestartSentinel(); if (!sentinel) { return; @@ -86,20 +85,15 @@ export async function scheduleRestartSentinelWake(params: { deps: CliDeps }) { (origin?.threadId != null ? String(origin.threadId) : undefined); try { - await agentCommand( - { - message, - sessionKey, - to: resolved.to, - channel, - deliver: true, - bestEffortDeliver: true, - messageChannel: channel, - threadId, - }, - defaultRuntime, - params.deps, - ); + await deliverOutboundPayloads({ + cfg, + channel, + to: resolved.to, + accountId: origin?.accountId, + threadId, + payloads: [{ text: message }], + bestEffort: true, + }); } catch (err) { enqueueSystemEvent(`${summary}\n${String(err)}`, { sessionKey }); } diff --git a/src/gateway/server.impl.ts b/src/gateway/server.impl.ts index 5b422a2bee4..7cc895df499 100644 --- a/src/gateway/server.impl.ts +++ b/src/gateway/server.impl.ts @@ -5,8 +5,10 @@ import type { RuntimeEnv } from "../runtime.js"; import type { ControlUiRootState } from "./control-ui.js"; import type { startBrowserControlServerIfEnabled } from "./server-browser.js"; import { resolveAgentWorkspaceDir, resolveDefaultAgentId } from "../agents/agent-scope.js"; +import { getActiveEmbeddedRunCount } from "../agents/pi-embedded-runner/runs.js"; import { registerSkillsChangeListener } from "../agents/skills/refresh.js"; import { initSubagentRegistry } from "../agents/subagent-registry.js"; +import { getTotalPendingReplies } from "../auto-reply/reply/dispatcher-registry.js"; import { type ChannelId, listChannelPlugins } from "../channels/plugins/index.js"; import { formatCliCommand } from "../cli/command-format.js"; import { createDefaultDeps } from "../cli/deps.js"; @@ -32,7 +34,7 @@ import { onHeartbeatEvent } from "../infra/heartbeat-events.js"; import { startHeartbeatRunner } from "../infra/heartbeat-runner.js"; import { getMachineDisplayName } from "../infra/machine-name.js"; import { ensureOpenClawCliOnPath } from "../infra/path-env.js"; -import { setGatewaySigusr1RestartPolicy } from "../infra/restart.js"; +import { setGatewaySigusr1RestartPolicy, setPreRestartDeferralCheck } from "../infra/restart.js"; import { primeRemoteSkillsCache, refreshRemoteBinsForConnectedNodes, @@ -42,6 +44,7 @@ import { scheduleGatewayUpdateCheck } from "../infra/update-startup.js"; import { startDiagnosticHeartbeat, stopDiagnosticHeartbeat } from "../logging/diagnostic.js"; import { createSubsystemLogger, runtimeForLogger } from "../logging/subsystem.js"; import { getGlobalHookRunner } from "../plugins/hook-runner-global.js"; +import { getTotalQueueSize } from "../process/command-queue.js"; import { runOnboardingWizard } from "../wizard/onboarding.js"; import { createAuthRateLimiter, type AuthRateLimiter } from "./auth-rate-limit.js"; import { startGatewayConfigReloader } from "./config-reload.js"; @@ -225,6 +228,9 @@ export async function startGatewayServer( startDiagnosticHeartbeat(); } setGatewaySigusr1RestartPolicy({ allowExternal: cfgAtStart.commands?.restart === true }); + setPreRestartDeferralCheck( + () => getTotalQueueSize() + getTotalPendingReplies() + getActiveEmbeddedRunCount(), + ); initSubagentRegistry(); const defaultAgentId = resolveDefaultAgentId(cfgAtStart); const defaultWorkspaceDir = resolveAgentWorkspaceDir(cfgAtStart, defaultAgentId); @@ -470,6 +476,18 @@ export async function startGatewayServer( void cron.start().catch((err) => logCron.error(`failed to start: ${String(err)}`)); + // Recover pending outbound deliveries from previous crash/restart. + void (async () => { + const { recoverPendingDeliveries } = await import("../infra/outbound/delivery-queue.js"); + const { deliverOutboundPayloads } = await import("../infra/outbound/deliver.js"); + const logRecovery = log.child("delivery-recovery"); + await recoverPendingDeliveries({ + deliver: deliverOutboundPayloads, + log: logRecovery, + cfg: cfgAtStart, + }); + })().catch((err) => log.error(`Delivery recovery failed: ${String(err)}`)); + const execApprovalManager = new ExecApprovalManager(); const execApprovalForwarder = createExecApprovalForwarder(); const execApprovalHandlers = createExecApprovalHandlers(execApprovalManager, { diff --git a/src/gateway/server.nodes.late-invoke.test.ts b/src/gateway/server.nodes.late-invoke.test.ts index b965e773464..8219b87842e 100644 --- a/src/gateway/server.nodes.late-invoke.test.ts +++ b/src/gateway/server.nodes.late-invoke.test.ts @@ -15,26 +15,25 @@ vi.mock("../infra/update-runner.js", () => ({ import { connectOk, + getFreePort, installGatewayTestHooks, rpcReq, - startServerWithClient, + startGatewayServer, } from "./test-helpers.js"; +import { testState } from "./test-helpers.mocks.js"; installGatewayTestHooks({ scope: "suite" }); -let server: Awaited>["server"]; -let ws: WebSocket; +let server: Awaited>; let port: number; let nodeWs: WebSocket; let nodeId: string; beforeAll(async () => { const token = "test-gateway-token-1234567890"; - const started = await startServerWithClient(token); - server = started.server; - ws = started.ws; - port = started.port; - await connectOk(ws, { token }); + testState.gatewayAuth = { mode: "token", token }; + port = await getFreePort(); + server = await startGatewayServer(port, { bind: "loopback" }); nodeWs = new WebSocket(`ws://127.0.0.1:${port}`); await new Promise((resolve) => nodeWs.once("open", resolve)); @@ -55,8 +54,7 @@ beforeAll(async () => { }); afterAll(async () => { - nodeWs.close(); - ws.close(); + nodeWs.terminate(); await server.close(); }); diff --git a/src/gateway/session-utils.ts b/src/gateway/session-utils.ts index fe13f78b0d0..178d4e7697d 100644 --- a/src/gateway/session-utils.ts +++ b/src/gateway/session-utils.ts @@ -394,7 +394,7 @@ export function listAgentsForGateway(cfg: OpenClawConfig): { let agentIds = listConfiguredAgentIds(cfg).filter((id) => allowedIds ? allowedIds.has(id) : true, ); - if (mainKey && !agentIds.includes(mainKey)) { + if (mainKey && !agentIds.includes(mainKey) && (!allowedIds || allowedIds.has(mainKey))) { agentIds = [...agentIds, mainKey]; } const agents = agentIds.map((id) => { diff --git a/src/gateway/test-helpers.server.ts b/src/gateway/test-helpers.server.ts index c58d2bb75c1..849e4243555 100644 --- a/src/gateway/test-helpers.server.ts +++ b/src/gateway/test-helpers.server.ts @@ -221,10 +221,10 @@ export function installGatewayTestHooks(options?: { scope?: "test" | "suite" }) if (scope === "suite") { beforeAll(async () => { await setupGatewayTestHome(); - await resetGatewayTestState({ uniqueConfigRoot: false }); + await resetGatewayTestState({ uniqueConfigRoot: true }); }); beforeEach(async () => { - await resetGatewayTestState({ uniqueConfigRoot: false }); + await resetGatewayTestState({ uniqueConfigRoot: true }); }, 60_000); afterEach(async () => { await cleanupGatewayTestHome({ restoreEnv: false }); diff --git a/src/gateway/tools-invoke-http.test.ts b/src/gateway/tools-invoke-http.test.ts index 0db60b71885..d373c274100 100644 --- a/src/gateway/tools-invoke-http.test.ts +++ b/src/gateway/tools-invoke-http.test.ts @@ -46,7 +46,7 @@ const invokeAgentsList = async (params: { } return await fetch(`http://127.0.0.1:${params.port}/tools/invoke`, { method: "POST", - headers: { "content-type": "application/json", ...params.headers }, + headers: { "content-type": "application/json", connection: "close", ...params.headers }, body: JSON.stringify(body), }); }; @@ -71,7 +71,7 @@ const invokeTool = async (params: { } return await fetch(`http://127.0.0.1:${params.port}/tools/invoke`, { method: "POST", - headers: { "content-type": "application/json", ...params.headers }, + headers: { "content-type": "application/json", connection: "close", ...params.headers }, body: JSON.stringify(body), }); }; @@ -144,41 +144,6 @@ describe("POST /tools/invoke", () => { expect(implicitBody.ok).toBe(true); }); - it("handles dedicated auth modes for password accept and token reject", async () => { - allowAgentsListForMain(); - - const passwordPort = await getFreePort(); - const passwordServer = await startGatewayServer(passwordPort, { - bind: "loopback", - auth: { mode: "password", password: "secret" }, - }); - try { - const passwordRes = await invokeAgentsList({ - port: passwordPort, - headers: { authorization: "Bearer secret" }, - sessionKey: "main", - }); - expect(passwordRes.status).toBe(200); - } finally { - await passwordServer.close(); - } - - const tokenPort = await getFreePort(); - const tokenServer = await startGatewayServer(tokenPort, { - bind: "loopback", - auth: { mode: "token", token: "t" }, - }); - try { - const tokenRes = await invokeAgentsList({ - port: tokenPort, - sessionKey: "main", - }); - expect(tokenRes.status).toBe(401); - } finally { - await tokenServer.close(); - } - }); - it("routes tools invoke before plugin HTTP handlers", async () => { const pluginHandler = vi.fn(async (_req: IncomingMessage, res: ServerResponse) => { res.statusCode = 418; diff --git a/src/hooks/bundled/README.md b/src/hooks/bundled/README.md index 4587d20a256..b3fb4e131a1 100644 --- a/src/hooks/bundled/README.md +++ b/src/hooks/bundled/README.md @@ -18,6 +18,20 @@ Automatically saves session context to memory when you issue `/new`. openclaw hooks enable session-memory ``` +### πŸ“Ž bootstrap-extra-files + +Injects extra bootstrap files (for example monorepo `AGENTS.md`/`TOOLS.md`) during prompt assembly. + +**Events**: `agent:bootstrap` +**What it does**: Expands configured workspace glob/path patterns and appends matching bootstrap files to injected context. +**Output**: No files written; context is modified in-memory only. + +**Enable**: + +```bash +openclaw hooks enable bootstrap-extra-files +``` + ### πŸ“ command-logger Logs all command events to a centralized audit file. diff --git a/src/hooks/bundled/bootstrap-extra-files/HOOK.md b/src/hooks/bundled/bootstrap-extra-files/HOOK.md new file mode 100644 index 00000000000..a46a07efd68 --- /dev/null +++ b/src/hooks/bundled/bootstrap-extra-files/HOOK.md @@ -0,0 +1,53 @@ +--- +name: bootstrap-extra-files +description: "Inject additional workspace bootstrap files via glob/path patterns" +homepage: https://docs.openclaw.ai/automation/hooks#bootstrap-extra-files +metadata: + { + "openclaw": + { + "emoji": "πŸ“Ž", + "events": ["agent:bootstrap"], + "requires": { "config": ["workspace.dir"] }, + "install": [{ "id": "bundled", "kind": "bundled", "label": "Bundled with OpenClaw" }], + }, + } +--- + +# Bootstrap Extra Files Hook + +Loads additional bootstrap files into `Project Context` during `agent:bootstrap`. + +## Why + +Use this when your workspace has multiple context roots (for example monorepos) and +you want to include extra `AGENTS.md`/`TOOLS.md`-class files without changing the +workspace root. + +## Configuration + +```json +{ + "hooks": { + "internal": { + "enabled": true, + "entries": { + "bootstrap-extra-files": { + "enabled": true, + "paths": ["packages/*/AGENTS.md", "packages/*/TOOLS.md"] + } + } + } + } +} +``` + +## Options + +- `paths` (string[]): preferred list of glob/path patterns. +- `patterns` (string[]): alias of `paths`. +- `files` (string[]): alias of `paths`. + +All paths are resolved from the workspace and must stay inside it (including realpath checks). +Only recognized bootstrap basenames are loaded (`AGENTS.md`, `SOUL.md`, `TOOLS.md`, +`IDENTITY.md`, `USER.md`, `HEARTBEAT.md`, `BOOTSTRAP.md`, `MEMORY.md`, `memory.md`). diff --git a/src/hooks/bundled/bootstrap-extra-files/handler.test.ts b/src/hooks/bundled/bootstrap-extra-files/handler.test.ts new file mode 100644 index 00000000000..2b945ad07a5 --- /dev/null +++ b/src/hooks/bundled/bootstrap-extra-files/handler.test.ts @@ -0,0 +1,106 @@ +import fs from "node:fs/promises"; +import path from "node:path"; +import { describe, expect, it } from "vitest"; +import type { OpenClawConfig } from "../../../config/config.js"; +import type { AgentBootstrapHookContext } from "../../hooks.js"; +import { makeTempWorkspace, writeWorkspaceFile } from "../../../test-helpers/workspace.js"; +import { createHookEvent } from "../../hooks.js"; +import handler from "./handler.js"; + +describe("bootstrap-extra-files hook", () => { + it("appends extra bootstrap files from configured patterns", async () => { + const tempDir = await makeTempWorkspace("openclaw-bootstrap-extra-"); + const extraDir = path.join(tempDir, "packages", "core"); + await fs.mkdir(extraDir, { recursive: true }); + await fs.writeFile(path.join(extraDir, "AGENTS.md"), "extra agents", "utf-8"); + + const cfg: OpenClawConfig = { + hooks: { + internal: { + entries: { + "bootstrap-extra-files": { + enabled: true, + paths: ["packages/*/AGENTS.md"], + }, + }, + }, + }, + }; + + const context: AgentBootstrapHookContext = { + workspaceDir: tempDir, + bootstrapFiles: [ + { + name: "AGENTS.md", + path: await writeWorkspaceFile({ + dir: tempDir, + name: "AGENTS.md", + content: "root agents", + }), + content: "root agents", + missing: false, + }, + ], + cfg, + sessionKey: "agent:main:main", + }; + + const event = createHookEvent("agent", "bootstrap", "agent:main:main", context); + await handler(event); + + const injected = context.bootstrapFiles.filter((f) => f.name === "AGENTS.md"); + expect(injected).toHaveLength(2); + expect(injected.some((f) => f.path.endsWith(path.join("packages", "core", "AGENTS.md")))).toBe( + true, + ); + }); + + it("re-applies subagent bootstrap allowlist after extras are added", async () => { + const tempDir = await makeTempWorkspace("openclaw-bootstrap-extra-subagent-"); + const extraDir = path.join(tempDir, "packages", "persona"); + await fs.mkdir(extraDir, { recursive: true }); + await fs.writeFile(path.join(extraDir, "SOUL.md"), "evil", "utf-8"); + + const cfg: OpenClawConfig = { + hooks: { + internal: { + entries: { + "bootstrap-extra-files": { + enabled: true, + paths: ["packages/*/SOUL.md"], + }, + }, + }, + }, + }; + + const context: AgentBootstrapHookContext = { + workspaceDir: tempDir, + bootstrapFiles: [ + { + name: "AGENTS.md", + path: await writeWorkspaceFile({ + dir: tempDir, + name: "AGENTS.md", + content: "root agents", + }), + content: "root agents", + missing: false, + }, + { + name: "TOOLS.md", + path: await writeWorkspaceFile({ dir: tempDir, name: "TOOLS.md", content: "root tools" }), + content: "root tools", + missing: false, + }, + ], + cfg, + sessionKey: "agent:main:subagent:abc", + }; + + const event = createHookEvent("agent", "bootstrap", "agent:main:subagent:abc", context); + await handler(event); + + expect(context.bootstrapFiles.map((f) => f.name).toSorted()).toEqual(["AGENTS.md", "TOOLS.md"]); + }); +}); diff --git a/src/hooks/bundled/bootstrap-extra-files/handler.ts b/src/hooks/bundled/bootstrap-extra-files/handler.ts new file mode 100644 index 00000000000..ada7286909d --- /dev/null +++ b/src/hooks/bundled/bootstrap-extra-files/handler.ts @@ -0,0 +1,59 @@ +import { + filterBootstrapFilesForSession, + loadExtraBootstrapFiles, +} from "../../../agents/workspace.js"; +import { resolveHookConfig } from "../../config.js"; +import { isAgentBootstrapEvent, type HookHandler } from "../../hooks.js"; + +const HOOK_KEY = "bootstrap-extra-files"; + +function normalizeStringArray(value: unknown): string[] { + if (!Array.isArray(value)) { + return []; + } + return value.map((v) => (typeof v === "string" ? v.trim() : "")).filter(Boolean); +} + +function resolveExtraBootstrapPatterns(hookConfig: Record): string[] { + const fromPaths = normalizeStringArray(hookConfig.paths); + if (fromPaths.length > 0) { + return fromPaths; + } + const fromPatterns = normalizeStringArray(hookConfig.patterns); + if (fromPatterns.length > 0) { + return fromPatterns; + } + return normalizeStringArray(hookConfig.files); +} + +const bootstrapExtraFilesHook: HookHandler = async (event) => { + if (!isAgentBootstrapEvent(event)) { + return; + } + + const context = event.context; + const hookConfig = resolveHookConfig(context.cfg, HOOK_KEY); + if (!hookConfig || hookConfig.enabled === false) { + return; + } + + const patterns = resolveExtraBootstrapPatterns(hookConfig as Record); + if (patterns.length === 0) { + return; + } + + try { + const extras = await loadExtraBootstrapFiles(context.workspaceDir, patterns); + if (extras.length === 0) { + return; + } + context.bootstrapFiles = filterBootstrapFilesForSession( + [...context.bootstrapFiles, ...extras], + context.sessionKey, + ); + } catch (err) { + console.warn(`[bootstrap-extra-files] failed: ${String(err)}`); + } +}; + +export default bootstrapExtraFilesHook; diff --git a/src/hooks/install.test.ts b/src/hooks/install.test.ts index 27a5616be27..0bbfc5bb6c8 100644 --- a/src/hooks/install.test.ts +++ b/src/hooks/install.test.ts @@ -4,28 +4,29 @@ import fs from "node:fs"; import os from "node:os"; import path from "node:path"; import * as tar from "tar"; -import { afterEach, describe, expect, it, vi } from "vitest"; +import { afterAll, describe, expect, it, vi } from "vitest"; -const tempDirs: string[] = []; +const fixtureRoot = path.join(os.tmpdir(), `openclaw-hook-install-${randomUUID()}`); +let tempDirIndex = 0; vi.mock("../process/exec.js", () => ({ runCommandWithTimeout: vi.fn(), })); function makeTempDir() { - const dir = path.join(os.tmpdir(), `openclaw-hook-install-${randomUUID()}`); + const dir = path.join(fixtureRoot, `case-${tempDirIndex++}`); fs.mkdirSync(dir, { recursive: true }); - tempDirs.push(dir); return dir; } -afterEach(() => { - for (const dir of tempDirs.splice(0)) { - try { - fs.rmSync(dir, { recursive: true, force: true }); - } catch { - // ignore cleanup failures - } +const { runCommandWithTimeout } = await import("../process/exec.js"); +const { installHooksFromArchive, installHooksFromPath } = await import("./install.js"); + +afterAll(() => { + try { + fs.rmSync(fixtureRoot, { recursive: true, force: true }); + } catch { + // ignore cleanup failures } }); @@ -61,7 +62,6 @@ describe("installHooksFromArchive", () => { fs.writeFileSync(archivePath, buffer); const hooksDir = path.join(stateDir, "hooks"); - const { installHooksFromArchive } = await import("./install.js"); const result = await installHooksFromArchive({ archivePath, hooksDir }); expect(result.ok).toBe(true); @@ -111,7 +111,6 @@ describe("installHooksFromArchive", () => { await tar.c({ cwd: workDir, file: archivePath }, ["package"]); const hooksDir = path.join(stateDir, "hooks"); - const { installHooksFromArchive } = await import("./install.js"); const result = await installHooksFromArchive({ archivePath, hooksDir }); expect(result.ok).toBe(true); @@ -160,7 +159,6 @@ describe("installHooksFromArchive", () => { await tar.c({ cwd: workDir, file: archivePath }, ["package"]); const hooksDir = path.join(stateDir, "hooks"); - const { installHooksFromArchive } = await import("./install.js"); const result = await installHooksFromArchive({ archivePath, hooksDir }); expect(result.ok).toBe(false); @@ -207,7 +205,6 @@ describe("installHooksFromArchive", () => { await tar.c({ cwd: workDir, file: archivePath }, ["package"]); const hooksDir = path.join(stateDir, "hooks"); - const { installHooksFromArchive } = await import("./install.js"); const result = await installHooksFromArchive({ archivePath, hooksDir }); expect(result.ok).toBe(false); @@ -253,11 +250,9 @@ describe("installHooksFromPath", () => { "utf-8", ); - const { runCommandWithTimeout } = await import("../process/exec.js"); const run = vi.mocked(runCommandWithTimeout); run.mockResolvedValue({ code: 0, stdout: "", stderr: "" }); - const { installHooksFromPath } = await import("./install.js"); const res = await installHooksFromPath({ path: pkgDir, hooksDir: path.join(stateDir, "hooks"), @@ -301,7 +296,6 @@ describe("installHooksFromPath", () => { fs.writeFileSync(path.join(hookDir, "handler.ts"), "export default async () => {};\n"); const hooksDir = path.join(stateDir, "hooks"); - const { installHooksFromPath } = await import("./install.js"); const result = await installHooksFromPath({ path: hookDir, hooksDir }); expect(result.ok).toBe(true); diff --git a/src/hooks/loader.test.ts b/src/hooks/loader.test.ts index 7bf4e11fa5b..bd1113fdaf3 100644 --- a/src/hooks/loader.test.ts +++ b/src/hooks/loader.test.ts @@ -1,7 +1,7 @@ import fs from "node:fs/promises"; import os from "node:os"; import path from "node:path"; -import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { afterEach, beforeEach, describe, expect, it } from "vitest"; import type { OpenClawConfig } from "../config/config.js"; import { clearInternalHooks, @@ -151,8 +151,6 @@ describe("loader", () => { }); it("should handle module loading errors gracefully", async () => { - const consoleError = vi.spyOn(console, "error").mockImplementation(() => {}); - const cfg: OpenClawConfig = { hooks: { internal: { @@ -167,19 +165,12 @@ describe("loader", () => { }, }; + // Should not throw and should return 0 (handler failed to load) const count = await loadInternalHooks(cfg, tmpDir); expect(count).toBe(0); - expect(consoleError).toHaveBeenCalledWith( - expect.stringContaining("Failed to load hook handler"), - expect.any(String), - ); - - consoleError.mockRestore(); }); it("should handle non-function exports", async () => { - const consoleError = vi.spyOn(console, "error").mockImplementation(() => {}); - // Create a module with a non-function export const handlerPath = path.join(tmpDir, "bad-export.js"); await fs.writeFile(handlerPath, 'export default "not a function";', "utf-8"); @@ -198,11 +189,9 @@ describe("loader", () => { }, }; + // Should not throw and should return 0 (handler is not a function) const count = await loadInternalHooks(cfg, tmpDir); expect(count).toBe(0); - expect(consoleError).toHaveBeenCalledWith(expect.stringContaining("is not a function")); - - consoleError.mockRestore(); }); it("should handle relative paths", async () => { diff --git a/src/hooks/loader.ts b/src/hooks/loader.ts index 9f558b8f6bf..9b6e854aa63 100644 --- a/src/hooks/loader.ts +++ b/src/hooks/loader.ts @@ -9,11 +9,14 @@ import path from "node:path"; import { pathToFileURL } from "node:url"; import type { OpenClawConfig } from "../config/config.js"; import type { InternalHookHandler } from "./internal-hooks.js"; +import { createSubsystemLogger } from "../logging/subsystem.js"; import { resolveHookConfig } from "./config.js"; import { shouldIncludeHook } from "./config.js"; import { registerInternalHook } from "./internal-hooks.js"; import { loadWorkspaceHookEntries } from "./workspace.js"; +const log = createSubsystemLogger("hooks:loader"); + /** * Load and register all hook handlers * @@ -78,16 +81,14 @@ export async function loadInternalHooks( const handler = mod[exportName]; if (typeof handler !== "function") { - console.error( - `Hook error: Handler '${exportName}' from ${entry.hook.name} is not a function`, - ); + log.error(`Handler '${exportName}' from ${entry.hook.name} is not a function`); continue; } // Register for all events listed in metadata const events = entry.metadata?.events ?? []; if (events.length === 0) { - console.warn(`Hook warning: Hook '${entry.hook.name}' has no events defined in metadata`); + log.warn(`Hook '${entry.hook.name}' has no events defined in metadata`); continue; } @@ -95,21 +96,19 @@ export async function loadInternalHooks( registerInternalHook(event, handler as InternalHookHandler); } - console.log( + log.info( `Registered hook: ${entry.hook.name} -> ${events.join(", ")}${exportName !== "default" ? ` (export: ${exportName})` : ""}`, ); loadedCount++; } catch (err) { - console.error( - `Failed to load hook ${entry.hook.name}:`, - err instanceof Error ? err.message : String(err), + log.error( + `Failed to load hook ${entry.hook.name}: ${err instanceof Error ? err.message : String(err)}`, ); } } } catch (err) { - console.error( - "Failed to load directory-based hooks:", - err instanceof Error ? err.message : String(err), + log.error( + `Failed to load directory-based hooks: ${err instanceof Error ? err.message : String(err)}`, ); } @@ -132,20 +131,18 @@ export async function loadInternalHooks( const handler = mod[exportName]; if (typeof handler !== "function") { - console.error(`Hook error: Handler '${exportName}' from ${modulePath} is not a function`); + log.error(`Handler '${exportName}' from ${modulePath} is not a function`); continue; } - // Register the handler registerInternalHook(handlerConfig.event, handler as InternalHookHandler); - console.log( + log.info( `Registered hook (legacy): ${handlerConfig.event} -> ${modulePath}${exportName !== "default" ? `#${exportName}` : ""}`, ); loadedCount++; } catch (err) { - console.error( - `Failed to load hook handler from ${handlerConfig.module}:`, - err instanceof Error ? err.message : String(err), + log.error( + `Failed to load hook handler from ${handlerConfig.module}: ${err instanceof Error ? err.message : String(err)}`, ); } } diff --git a/src/imessage/monitor/monitor-provider.ts b/src/imessage/monitor/monitor-provider.ts index a9e0d93f7cc..445fe73aeae 100644 --- a/src/imessage/monitor/monitor-provider.ts +++ b/src/imessage/monitor/monitor-provider.ts @@ -659,6 +659,7 @@ export async function monitorIMessageProvider(opts: MonitorIMessageOpts = {}): P onModelSelected, }, }); + if (!queuedFinal) { if (isGroup && historyKey) { clearHistoryEntriesIfEnabled({ diff --git a/src/imessage/monitor/runtime.ts b/src/imessage/monitor/runtime.ts index 67ee2e4ac44..004fc2fdd9c 100644 --- a/src/imessage/monitor/runtime.ts +++ b/src/imessage/monitor/runtime.ts @@ -1,16 +1,8 @@ -import type { RuntimeEnv } from "../../runtime.js"; import type { MonitorIMessageOpts } from "./types.js"; +import { createNonExitingRuntime, type RuntimeEnv } from "../../runtime.js"; export function resolveRuntime(opts: MonitorIMessageOpts): RuntimeEnv { - return ( - opts.runtime ?? { - log: console.log, - error: console.error, - exit: (code: number): never => { - throw new Error(`exit ${code}`); - }, - } - ); + return opts.runtime ?? createNonExitingRuntime(); } export function normalizeAllowList(list?: Array) { diff --git a/src/infra/gateway-lock.test.ts b/src/infra/gateway-lock.test.ts index 12a93fd5857..3b19f25dda8 100644 --- a/src/infra/gateway-lock.test.ts +++ b/src/infra/gateway-lock.test.ts @@ -3,12 +3,16 @@ import fsSync from "node:fs"; import fs from "node:fs/promises"; import os from "node:os"; import path from "node:path"; -import { describe, expect, it, vi } from "vitest"; +import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; import { resolveConfigPath, resolveGatewayLockDir, resolveStateDir } from "../config/paths.js"; import { acquireGatewayLock, GatewayLockError } from "./gateway-lock.js"; +let fixtureRoot = ""; +let fixtureCount = 0; + async function makeEnv() { - const dir = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-gateway-lock-")); + const dir = path.join(fixtureRoot, `case-${fixtureCount++}`); + await fs.mkdir(dir, { recursive: true }); const configPath = path.join(dir, "openclaw.json"); await fs.writeFile(configPath, "{}", "utf8"); await fs.mkdir(resolveGatewayLockDir(), { recursive: true }); @@ -18,9 +22,7 @@ async function makeEnv() { OPENCLAW_STATE_DIR: dir, OPENCLAW_CONFIG_PATH: configPath, }, - cleanup: async () => { - await fs.rm(dir, { recursive: true, force: true }); - }, + cleanup: async () => {}, }; } @@ -61,13 +63,21 @@ function makeProcStat(pid: number, startTime: number) { } describe("gateway lock", () => { + beforeAll(async () => { + fixtureRoot = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-gateway-lock-")); + }); + + afterAll(async () => { + await fs.rm(fixtureRoot, { recursive: true, force: true }); + }); + it("blocks concurrent acquisition until release", async () => { const { env, cleanup } = await makeEnv(); const lock = await acquireGatewayLock({ env, allowInTests: true, - timeoutMs: 200, - pollIntervalMs: 20, + timeoutMs: 80, + pollIntervalMs: 5, }); expect(lock).not.toBeNull(); @@ -75,8 +85,8 @@ describe("gateway lock", () => { acquireGatewayLock({ env, allowInTests: true, - timeoutMs: 200, - pollIntervalMs: 20, + timeoutMs: 80, + pollIntervalMs: 5, }), ).rejects.toBeInstanceOf(GatewayLockError); @@ -84,8 +94,8 @@ describe("gateway lock", () => { const lock2 = await acquireGatewayLock({ env, allowInTests: true, - timeoutMs: 200, - pollIntervalMs: 20, + timeoutMs: 80, + pollIntervalMs: 5, }); await lock2?.release(); await cleanup(); @@ -114,8 +124,8 @@ describe("gateway lock", () => { const lock = await acquireGatewayLock({ env, allowInTests: true, - timeoutMs: 200, - pollIntervalMs: 20, + timeoutMs: 80, + pollIntervalMs: 5, platform: "linux", }); expect(lock).not.toBeNull(); @@ -148,8 +158,8 @@ describe("gateway lock", () => { acquireGatewayLock({ env, allowInTests: true, - timeoutMs: 120, - pollIntervalMs: 20, + timeoutMs: 50, + pollIntervalMs: 5, staleMs: 10_000, platform: "linux", }), @@ -173,8 +183,8 @@ describe("gateway lock", () => { const lock = await acquireGatewayLock({ env, allowInTests: true, - timeoutMs: 200, - pollIntervalMs: 20, + timeoutMs: 80, + pollIntervalMs: 5, staleMs: 1, platform: "linux", }); diff --git a/src/infra/heartbeat-runner.returns-default-unset.test.ts b/src/infra/heartbeat-runner.returns-default-unset.test.ts index 687ea5dbf28..90359fadacb 100644 --- a/src/infra/heartbeat-runner.returns-default-unset.test.ts +++ b/src/infra/heartbeat-runner.returns-default-unset.test.ts @@ -1020,6 +1020,142 @@ describe("runHeartbeatOnce", () => { } }); + it("does not skip wake-triggered heartbeat when HEARTBEAT.md is effectively empty", async () => { + const tmpDir = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-hb-")); + const storePath = path.join(tmpDir, "sessions.json"); + const workspaceDir = path.join(tmpDir, "workspace"); + const replySpy = vi.spyOn(replyModule, "getReplyFromConfig"); + try { + await fs.mkdir(workspaceDir, { recursive: true }); + await fs.writeFile( + path.join(workspaceDir, "HEARTBEAT.md"), + "# HEARTBEAT.md\n\n## Tasks\n\n", + "utf-8", + ); + + const cfg: OpenClawConfig = { + agents: { + defaults: { + workspace: workspaceDir, + heartbeat: { every: "5m", target: "whatsapp" }, + }, + }, + channels: { whatsapp: { allowFrom: ["*"] } }, + session: { store: storePath }, + }; + const sessionKey = resolveMainSessionKey(cfg); + + await fs.writeFile( + storePath, + JSON.stringify( + { + [sessionKey]: { + sessionId: "sid", + updatedAt: Date.now(), + lastChannel: "whatsapp", + lastTo: "+1555", + }, + }, + null, + 2, + ), + ); + + replySpy.mockResolvedValue({ text: "wake event processed" }); + const sendWhatsApp = vi.fn().mockResolvedValue({ + messageId: "m1", + toJid: "jid", + }); + + const res = await runHeartbeatOnce({ + cfg, + reason: "wake", + deps: { + sendWhatsApp, + getQueueSize: () => 0, + nowMs: () => 0, + webAuthExists: async () => true, + hasActiveWebListener: () => true, + }, + }); + + expect(res.status).toBe("ran"); + expect(replySpy).toHaveBeenCalled(); + expect(sendWhatsApp).toHaveBeenCalledTimes(1); + } finally { + replySpy.mockRestore(); + await fs.rm(tmpDir, { recursive: true, force: true }); + } + }); + + it("does not skip hook-triggered heartbeat when HEARTBEAT.md is effectively empty", async () => { + const tmpDir = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-hb-")); + const storePath = path.join(tmpDir, "sessions.json"); + const workspaceDir = path.join(tmpDir, "workspace"); + const replySpy = vi.spyOn(replyModule, "getReplyFromConfig"); + try { + await fs.mkdir(workspaceDir, { recursive: true }); + await fs.writeFile( + path.join(workspaceDir, "HEARTBEAT.md"), + "# HEARTBEAT.md\n\n## Tasks\n\n", + "utf-8", + ); + + const cfg: OpenClawConfig = { + agents: { + defaults: { + workspace: workspaceDir, + heartbeat: { every: "5m", target: "whatsapp" }, + }, + }, + channels: { whatsapp: { allowFrom: ["*"] } }, + session: { store: storePath }, + }; + const sessionKey = resolveMainSessionKey(cfg); + + await fs.writeFile( + storePath, + JSON.stringify( + { + [sessionKey]: { + sessionId: "sid", + updatedAt: Date.now(), + lastChannel: "whatsapp", + lastTo: "+1555", + }, + }, + null, + 2, + ), + ); + + replySpy.mockResolvedValue({ text: "hook event processed" }); + const sendWhatsApp = vi.fn().mockResolvedValue({ + messageId: "m1", + toJid: "jid", + }); + + const res = await runHeartbeatOnce({ + cfg, + reason: "hook:wake", + deps: { + sendWhatsApp, + getQueueSize: () => 0, + nowMs: () => 0, + webAuthExists: async () => true, + hasActiveWebListener: () => true, + }, + }); + + expect(res.status).toBe("ran"); + expect(replySpy).toHaveBeenCalled(); + expect(sendWhatsApp).toHaveBeenCalledTimes(1); + } finally { + replySpy.mockRestore(); + await fs.rm(tmpDir, { recursive: true, force: true }); + } + }); + it("runs heartbeat when HEARTBEAT.md has actionable content", async () => { const tmpDir = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-hb-")); const storePath = path.join(tmpDir, "sessions.json"); diff --git a/src/infra/heartbeat-runner.ts b/src/infra/heartbeat-runner.ts index d90a978bde6..9ee8bcc09f6 100644 --- a/src/infra/heartbeat-runner.ts +++ b/src/infra/heartbeat-runner.ts @@ -426,10 +426,11 @@ export async function runHeartbeatOnce(opts: { // Skip heartbeat if HEARTBEAT.md exists but has no actionable content. // This saves API calls/costs when the file is effectively empty (only comments/headers). - // EXCEPTION: Don't skip for exec events or cron events - they have pending system events - // to process regardless of HEARTBEAT.md content. + // EXCEPTION: Don't skip for exec events, cron events, or explicit wake requests - + // they have pending system events to process regardless of HEARTBEAT.md content. const isExecEventReason = opts.reason === "exec-event"; const isCronEventReason = Boolean(opts.reason?.startsWith("cron:")); + const isWakeReason = opts.reason === "wake" || Boolean(opts.reason?.startsWith("hook:")); const workspaceDir = resolveAgentWorkspaceDir(cfg, agentId); const heartbeatFilePath = path.join(workspaceDir, DEFAULT_HEARTBEAT_FILENAME); try { @@ -437,7 +438,8 @@ export async function runHeartbeatOnce(opts: { if ( isHeartbeatContentEffectivelyEmpty(heartbeatFileContent) && !isExecEventReason && - !isCronEventReason + !isCronEventReason && + !isWakeReason ) { emitHeartbeatEvent({ status: "skipped", diff --git a/src/infra/heartbeat-wake.test.ts b/src/infra/heartbeat-wake.test.ts index b3f8e0d32f7..63d47523023 100644 --- a/src/infra/heartbeat-wake.test.ts +++ b/src/infra/heartbeat-wake.test.ts @@ -173,6 +173,59 @@ describe("heartbeat-wake", () => { expect(handler).toHaveBeenCalledWith({ reason: "exec-event" }); }); + it("resets running/scheduled flags when new handler is registered", async () => { + vi.useFakeTimers(); + + // Simulate a handler that's mid-execution when SIGUSR1 fires. + // We do this by having the handler hang forever (never resolve). + let resolveHang: () => void; + const hangPromise = new Promise((r) => { + resolveHang = r; + }); + const handlerA = vi + .fn() + .mockReturnValue(hangPromise.then(() => ({ status: "ran" as const, durationMs: 1 }))); + setHeartbeatWakeHandler(handlerA); + + // Trigger the handler β€” it starts running but never finishes + requestHeartbeatNow({ reason: "interval", coalesceMs: 0 }); + await vi.advanceTimersByTimeAsync(1); + expect(handlerA).toHaveBeenCalledTimes(1); + + // Now simulate SIGUSR1: register a new handler while handlerA is still running. + // Without the fix, `running` would stay true and handlerB would never fire. + const handlerB = vi.fn().mockResolvedValue({ status: "ran", durationMs: 1 }); + setHeartbeatWakeHandler(handlerB); + + // handlerB should be able to fire (running was reset) + requestHeartbeatNow({ reason: "interval", coalesceMs: 0 }); + await vi.advanceTimersByTimeAsync(1); + expect(handlerB).toHaveBeenCalledTimes(1); + + // Clean up the hanging promise + resolveHang!(); + await Promise.resolve(); + }); + + it("clears stale retry cooldown when a new handler is registered", async () => { + vi.useFakeTimers(); + const handlerA = vi.fn().mockResolvedValue({ status: "skipped", reason: "requests-in-flight" }); + setHeartbeatWakeHandler(handlerA); + + requestHeartbeatNow({ reason: "interval", coalesceMs: 0 }); + await vi.advanceTimersByTimeAsync(1); + expect(handlerA).toHaveBeenCalledTimes(1); + + // Simulate SIGUSR1 startup with a fresh wake handler. + const handlerB = vi.fn().mockResolvedValue({ status: "ran", durationMs: 1 }); + setHeartbeatWakeHandler(handlerB); + + requestHeartbeatNow({ reason: "manual", coalesceMs: 0 }); + await vi.advanceTimersByTimeAsync(1); + expect(handlerB).toHaveBeenCalledTimes(1); + expect(handlerB).toHaveBeenCalledWith({ reason: "manual" }); + }); + it("drains pending wake once a handler is registered", async () => { vi.useFakeTimers(); diff --git a/src/infra/heartbeat-wake.ts b/src/infra/heartbeat-wake.ts index 72f97378f67..6297b5ffb68 100644 --- a/src/infra/heartbeat-wake.ts +++ b/src/infra/heartbeat-wake.ts @@ -146,6 +146,23 @@ export function setHeartbeatWakeHandler(next: HeartbeatWakeHandler | null): () = handlerGeneration += 1; const generation = handlerGeneration; handler = next; + if (next) { + // New lifecycle starting (e.g. after SIGUSR1 in-process restart). + // Clear any timer metadata from the previous lifecycle so stale retry + // cooldowns do not delay a fresh handler. + if (timer) { + clearTimeout(timer); + } + timer = null; + timerDueAt = null; + timerKind = null; + // Reset module-level execution state that may be stale from interrupted + // runs in the previous lifecycle. Without this, `running === true` from + // an interrupted heartbeat blocks all future schedule() attempts, and + // `scheduled === true` can cause spurious immediate re-runs. + running = false; + scheduled = false; + } if (handler && pendingWake) { schedule(DEFAULT_COALESCE_MS, "normal"); } diff --git a/src/infra/infra-runtime.test.ts b/src/infra/infra-runtime.test.ts index 926c1f224c6..78e6d15f9a3 100644 --- a/src/infra/infra-runtime.test.ts +++ b/src/infra/infra-runtime.test.ts @@ -6,9 +6,12 @@ import { ensureBinary } from "./binaries.js"; import { __testing, consumeGatewaySigusr1RestartAuthorization, + emitGatewayRestart, isGatewaySigusr1RestartExternallyAllowed, + markGatewaySigusr1RestartHandled, scheduleGatewaySigusr1Restart, setGatewaySigusr1RestartPolicy, + setPreRestartDeferralCheck, } from "./restart.js"; import { createTelegramRetryRunner } from "./retry-policy.js"; import { getShellPathFromLoginShell, resetShellPathCacheForTests } from "./shell-env.js"; @@ -79,11 +82,15 @@ describe("infra runtime", () => { __testing.resetSigusr1State(); }); - it("consumes a scheduled authorization once", async () => { + it("authorizes exactly once when scheduled restart emits", async () => { expect(consumeGatewaySigusr1RestartAuthorization()).toBe(false); scheduleGatewaySigusr1Restart({ delayMs: 0 }); + // No pre-authorization before the scheduled emission fires. + expect(consumeGatewaySigusr1RestartAuthorization()).toBe(false); + await vi.advanceTimersByTimeAsync(0); + expect(consumeGatewaySigusr1RestartAuthorization()).toBe(true); expect(consumeGatewaySigusr1RestartAuthorization()).toBe(false); @@ -95,6 +102,129 @@ describe("infra runtime", () => { setGatewaySigusr1RestartPolicy({ allowExternal: true }); expect(isGatewaySigusr1RestartExternallyAllowed()).toBe(true); }); + + it("suppresses duplicate emit until the restart cycle is marked handled", () => { + const emitSpy = vi.spyOn(process, "emit"); + const handler = () => {}; + process.on("SIGUSR1", handler); + try { + expect(emitGatewayRestart()).toBe(true); + expect(emitGatewayRestart()).toBe(false); + expect(consumeGatewaySigusr1RestartAuthorization()).toBe(true); + + markGatewaySigusr1RestartHandled(); + + expect(emitGatewayRestart()).toBe(true); + const sigusr1Emits = emitSpy.mock.calls.filter((args) => args[0] === "SIGUSR1"); + expect(sigusr1Emits.length).toBe(2); + } finally { + process.removeListener("SIGUSR1", handler); + } + }); + }); + + describe("pre-restart deferral check", () => { + beforeEach(() => { + __testing.resetSigusr1State(); + vi.useFakeTimers(); + vi.spyOn(process, "kill").mockImplementation(() => true); + }); + + afterEach(async () => { + await vi.runOnlyPendingTimersAsync(); + vi.useRealTimers(); + vi.restoreAllMocks(); + __testing.resetSigusr1State(); + }); + + it("emits SIGUSR1 immediately when no deferral check is registered", async () => { + const emitSpy = vi.spyOn(process, "emit"); + const handler = () => {}; + process.on("SIGUSR1", handler); + try { + scheduleGatewaySigusr1Restart({ delayMs: 0 }); + await vi.advanceTimersByTimeAsync(0); + expect(emitSpy).toHaveBeenCalledWith("SIGUSR1"); + } finally { + process.removeListener("SIGUSR1", handler); + } + }); + + it("emits SIGUSR1 immediately when deferral check returns 0", async () => { + const emitSpy = vi.spyOn(process, "emit"); + const handler = () => {}; + process.on("SIGUSR1", handler); + try { + setPreRestartDeferralCheck(() => 0); + scheduleGatewaySigusr1Restart({ delayMs: 0 }); + await vi.advanceTimersByTimeAsync(0); + expect(emitSpy).toHaveBeenCalledWith("SIGUSR1"); + } finally { + process.removeListener("SIGUSR1", handler); + } + }); + + it("defers SIGUSR1 until deferral check returns 0", async () => { + const emitSpy = vi.spyOn(process, "emit"); + const handler = () => {}; + process.on("SIGUSR1", handler); + try { + let pending = 2; + setPreRestartDeferralCheck(() => pending); + scheduleGatewaySigusr1Restart({ delayMs: 0 }); + + // After initial delay fires, deferral check returns 2 β€” should NOT emit yet + await vi.advanceTimersByTimeAsync(0); + expect(emitSpy).not.toHaveBeenCalledWith("SIGUSR1"); + + // After one poll (500ms), still pending + await vi.advanceTimersByTimeAsync(500); + expect(emitSpy).not.toHaveBeenCalledWith("SIGUSR1"); + + // Drain pending work + pending = 0; + await vi.advanceTimersByTimeAsync(500); + expect(emitSpy).toHaveBeenCalledWith("SIGUSR1"); + } finally { + process.removeListener("SIGUSR1", handler); + } + }); + + it("emits SIGUSR1 after deferral timeout even if still pending", async () => { + const emitSpy = vi.spyOn(process, "emit"); + const handler = () => {}; + process.on("SIGUSR1", handler); + try { + setPreRestartDeferralCheck(() => 5); // always pending + scheduleGatewaySigusr1Restart({ delayMs: 0 }); + + // Fire initial timeout + await vi.advanceTimersByTimeAsync(0); + expect(emitSpy).not.toHaveBeenCalledWith("SIGUSR1"); + + // Advance past the 30s max deferral wait + await vi.advanceTimersByTimeAsync(30_000); + expect(emitSpy).toHaveBeenCalledWith("SIGUSR1"); + } finally { + process.removeListener("SIGUSR1", handler); + } + }); + + it("emits SIGUSR1 if deferral check throws", async () => { + const emitSpy = vi.spyOn(process, "emit"); + const handler = () => {}; + process.on("SIGUSR1", handler); + try { + setPreRestartDeferralCheck(() => { + throw new Error("boom"); + }); + scheduleGatewaySigusr1Restart({ delayMs: 0 }); + await vi.advanceTimersByTimeAsync(0); + expect(emitSpy).toHaveBeenCalledWith("SIGUSR1"); + } finally { + process.removeListener("SIGUSR1", handler); + } + }); }); describe("getShellPathFromLoginShell", () => { diff --git a/src/infra/outbound/deliver.test.ts b/src/infra/outbound/deliver.test.ts index 221050cc49d..3247149bec4 100644 --- a/src/infra/outbound/deliver.test.ts +++ b/src/infra/outbound/deliver.test.ts @@ -20,6 +20,11 @@ const hookMocks = vi.hoisted(() => ({ runMessageSent: vi.fn(async () => {}), }, })); +const queueMocks = vi.hoisted(() => ({ + enqueueDelivery: vi.fn(async () => "mock-queue-id"), + ackDelivery: vi.fn(async () => {}), + failDelivery: vi.fn(async () => {}), +})); vi.mock("../../config/sessions.js", async () => { const actual = await vi.importActual( @@ -33,6 +38,11 @@ vi.mock("../../config/sessions.js", async () => { vi.mock("../../plugins/hook-runner-global.js", () => ({ getGlobalHookRunner: () => hookMocks.runner, })); +vi.mock("./delivery-queue.js", () => ({ + enqueueDelivery: queueMocks.enqueueDelivery, + ackDelivery: queueMocks.ackDelivery, + failDelivery: queueMocks.failDelivery, +})); const { deliverOutboundPayloads, normalizeOutboundPayloads } = await import("./deliver.js"); @@ -43,6 +53,12 @@ describe("deliverOutboundPayloads", () => { hookMocks.runner.hasHooks.mockReturnValue(false); hookMocks.runner.runMessageSent.mockReset(); hookMocks.runner.runMessageSent.mockResolvedValue(undefined); + queueMocks.enqueueDelivery.mockReset(); + queueMocks.enqueueDelivery.mockResolvedValue("mock-queue-id"); + queueMocks.ackDelivery.mockReset(); + queueMocks.ackDelivery.mockResolvedValue(undefined); + queueMocks.failDelivery.mockReset(); + queueMocks.failDelivery.mockResolvedValue(undefined); }); afterEach(() => { @@ -389,6 +405,57 @@ describe("deliverOutboundPayloads", () => { expect(results).toEqual([{ channel: "whatsapp", messageId: "w2", toJid: "jid" }]); }); + it("calls failDelivery instead of ackDelivery on bestEffort partial failure", async () => { + const sendWhatsApp = vi + .fn() + .mockRejectedValueOnce(new Error("fail")) + .mockResolvedValueOnce({ messageId: "w2", toJid: "jid" }); + const onError = vi.fn(); + const cfg: OpenClawConfig = {}; + + await deliverOutboundPayloads({ + cfg, + channel: "whatsapp", + to: "+1555", + payloads: [{ text: "a" }, { text: "b" }], + deps: { sendWhatsApp }, + bestEffort: true, + onError, + }); + + // onError was called for the first payload's failure. + expect(onError).toHaveBeenCalledTimes(1); + + // Queue entry should NOT be acked β€” failDelivery should be called instead. + expect(queueMocks.ackDelivery).not.toHaveBeenCalled(); + expect(queueMocks.failDelivery).toHaveBeenCalledWith( + "mock-queue-id", + "partial delivery failure (bestEffort)", + ); + }); + + it("acks the queue entry when delivery is aborted", async () => { + const sendWhatsApp = vi.fn().mockResolvedValue({ messageId: "w1", toJid: "jid" }); + const abortController = new AbortController(); + abortController.abort(); + const cfg: OpenClawConfig = {}; + + await expect( + deliverOutboundPayloads({ + cfg, + channel: "whatsapp", + to: "+1555", + payloads: [{ text: "a" }], + deps: { sendWhatsApp }, + abortSignal: abortController.signal, + }), + ).rejects.toThrow("Operation aborted"); + + expect(queueMocks.ackDelivery).toHaveBeenCalledWith("mock-queue-id"); + expect(queueMocks.failDelivery).not.toHaveBeenCalled(); + expect(sendWhatsApp).not.toHaveBeenCalled(); + }); + it("passes normalized payload to onError", async () => { const sendWhatsApp = vi.fn().mockRejectedValue(new Error("boom")); const onError = vi.fn(); diff --git a/src/infra/outbound/deliver.ts b/src/infra/outbound/deliver.ts index 6460efc01a0..acbd4936907 100644 --- a/src/infra/outbound/deliver.ts +++ b/src/infra/outbound/deliver.ts @@ -25,6 +25,7 @@ import { getGlobalHookRunner } from "../../plugins/hook-runner-global.js"; import { markdownToSignalTextChunks, type SignalTextStyleRange } from "../../signal/format.js"; import { sendMessageSignal } from "../../signal/send.js"; import { throwIfAborted } from "./abort.js"; +import { ackDelivery, enqueueDelivery, failDelivery } from "./delivery-queue.js"; import { normalizeReplyPayloadsForDelivery } from "./payloads.js"; export type { NormalizedOutboundPayload } from "./payloads.js"; @@ -178,6 +179,8 @@ function createPluginHandler(params: { }; } +const isAbortError = (err: unknown): boolean => err instanceof Error && err.name === "AbortError"; + export async function deliverOutboundPayloads(params: { cfg: OpenClawConfig; channel: Exclude; @@ -199,6 +202,88 @@ export async function deliverOutboundPayloads(params: { mediaUrls?: string[]; }; silent?: boolean; + /** @internal Skip write-ahead queue (used by crash-recovery to avoid re-enqueueing). */ + skipQueue?: boolean; +}): Promise { + const { channel, to, payloads } = params; + + // Write-ahead delivery queue: persist before sending, remove after success. + const queueId = params.skipQueue + ? null + : await enqueueDelivery({ + channel, + to, + accountId: params.accountId, + payloads, + threadId: params.threadId, + replyToId: params.replyToId, + bestEffort: params.bestEffort, + gifPlayback: params.gifPlayback, + silent: params.silent, + mirror: params.mirror, + }).catch(() => null); // Best-effort β€” don't block delivery if queue write fails. + + // Wrap onError to detect partial failures under bestEffort mode. + // When bestEffort is true, per-payload errors are caught and passed to onError + // without throwing β€” so the outer try/catch never fires. We track whether any + // payload failed so we can call failDelivery instead of ackDelivery. + let hadPartialFailure = false; + const wrappedParams = params.onError + ? { + ...params, + onError: (err: unknown, payload: NormalizedOutboundPayload) => { + hadPartialFailure = true; + params.onError!(err, payload); + }, + } + : params; + + try { + const results = await deliverOutboundPayloadsCore(wrappedParams); + if (queueId) { + if (hadPartialFailure) { + await failDelivery(queueId, "partial delivery failure (bestEffort)").catch(() => {}); + } else { + await ackDelivery(queueId).catch(() => {}); // Best-effort cleanup. + } + } + return results; + } catch (err) { + if (queueId) { + if (isAbortError(err)) { + await ackDelivery(queueId).catch(() => {}); + } else { + await failDelivery(queueId, err instanceof Error ? err.message : String(err)).catch( + () => {}, + ); + } + } + throw err; + } +} + +/** Core delivery logic (extracted for queue wrapper). */ +async function deliverOutboundPayloadsCore(params: { + cfg: OpenClawConfig; + channel: Exclude; + to: string; + accountId?: string; + payloads: ReplyPayload[]; + replyToId?: string | null; + threadId?: string | number | null; + deps?: OutboundSendDeps; + gifPlayback?: boolean; + abortSignal?: AbortSignal; + bestEffort?: boolean; + onError?: (err: unknown, payload: NormalizedOutboundPayload) => void; + onPayload?: (payload: NormalizedOutboundPayload) => void; + mirror?: { + sessionKey: string; + agentId?: string; + text?: string; + mediaUrls?: string[]; + }; + silent?: boolean; }): Promise { const { cfg, channel, to, payloads } = params; const accountId = params.accountId; diff --git a/src/infra/outbound/delivery-queue.test.ts b/src/infra/outbound/delivery-queue.test.ts new file mode 100644 index 00000000000..ee94d13b62b --- /dev/null +++ b/src/infra/outbound/delivery-queue.test.ts @@ -0,0 +1,373 @@ +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { + ackDelivery, + computeBackoffMs, + enqueueDelivery, + failDelivery, + loadPendingDeliveries, + MAX_RETRIES, + moveToFailed, + recoverPendingDeliveries, +} from "./delivery-queue.js"; + +let tmpDir: string; + +beforeEach(() => { + tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), "openclaw-dq-test-")); +}); + +afterEach(() => { + fs.rmSync(tmpDir, { recursive: true, force: true }); +}); + +describe("enqueue + ack lifecycle", () => { + it("creates and removes a queue entry", async () => { + const id = await enqueueDelivery( + { + channel: "whatsapp", + to: "+1555", + payloads: [{ text: "hello" }], + bestEffort: true, + gifPlayback: true, + silent: true, + mirror: { + sessionKey: "agent:main:main", + text: "hello", + mediaUrls: ["https://example.com/file.png"], + }, + }, + tmpDir, + ); + + // Entry file exists after enqueue. + const queueDir = path.join(tmpDir, "delivery-queue"); + const files = fs.readdirSync(queueDir).filter((f) => f.endsWith(".json")); + expect(files).toHaveLength(1); + expect(files[0]).toBe(`${id}.json`); + + // Entry contents are correct. + const entry = JSON.parse(fs.readFileSync(path.join(queueDir, files[0]), "utf-8")); + expect(entry).toMatchObject({ + id, + channel: "whatsapp", + to: "+1555", + bestEffort: true, + gifPlayback: true, + silent: true, + mirror: { + sessionKey: "agent:main:main", + text: "hello", + mediaUrls: ["https://example.com/file.png"], + }, + retryCount: 0, + }); + expect(entry.payloads).toEqual([{ text: "hello" }]); + + // Ack removes the file. + await ackDelivery(id, tmpDir); + const remaining = fs.readdirSync(queueDir).filter((f) => f.endsWith(".json")); + expect(remaining).toHaveLength(0); + }); + + it("ack is idempotent (no error on missing file)", async () => { + await expect(ackDelivery("nonexistent-id", tmpDir)).resolves.toBeUndefined(); + }); +}); + +describe("failDelivery", () => { + it("increments retryCount and sets lastError", async () => { + const id = await enqueueDelivery( + { + channel: "telegram", + to: "123", + payloads: [{ text: "test" }], + }, + tmpDir, + ); + + await failDelivery(id, "connection refused", tmpDir); + + const queueDir = path.join(tmpDir, "delivery-queue"); + const entry = JSON.parse(fs.readFileSync(path.join(queueDir, `${id}.json`), "utf-8")); + expect(entry.retryCount).toBe(1); + expect(entry.lastError).toBe("connection refused"); + }); +}); + +describe("moveToFailed", () => { + it("moves entry to failed/ subdirectory", async () => { + const id = await enqueueDelivery( + { + channel: "slack", + to: "#general", + payloads: [{ text: "hi" }], + }, + tmpDir, + ); + + await moveToFailed(id, tmpDir); + + const queueDir = path.join(tmpDir, "delivery-queue"); + const failedDir = path.join(queueDir, "failed"); + expect(fs.existsSync(path.join(queueDir, `${id}.json`))).toBe(false); + expect(fs.existsSync(path.join(failedDir, `${id}.json`))).toBe(true); + }); +}); + +describe("loadPendingDeliveries", () => { + it("returns empty array when queue directory does not exist", async () => { + const nonexistent = path.join(tmpDir, "no-such-dir"); + const entries = await loadPendingDeliveries(nonexistent); + expect(entries).toEqual([]); + }); + + it("loads multiple entries", async () => { + await enqueueDelivery({ channel: "whatsapp", to: "+1", payloads: [{ text: "a" }] }, tmpDir); + await enqueueDelivery({ channel: "telegram", to: "2", payloads: [{ text: "b" }] }, tmpDir); + + const entries = await loadPendingDeliveries(tmpDir); + expect(entries).toHaveLength(2); + }); +}); + +describe("computeBackoffMs", () => { + it("returns 0 for retryCount 0", () => { + expect(computeBackoffMs(0)).toBe(0); + }); + + it("returns correct backoff for each retry", () => { + expect(computeBackoffMs(1)).toBe(5_000); + expect(computeBackoffMs(2)).toBe(25_000); + expect(computeBackoffMs(3)).toBe(120_000); + expect(computeBackoffMs(4)).toBe(600_000); + // Beyond defined schedule β€” clamps to last value. + expect(computeBackoffMs(5)).toBe(600_000); + }); +}); + +describe("recoverPendingDeliveries", () => { + const noopDelay = async () => {}; + const baseCfg = {}; + + it("recovers entries from a simulated crash", async () => { + // Manually create two queue entries as if gateway crashed before delivery. + await enqueueDelivery({ channel: "whatsapp", to: "+1", payloads: [{ text: "a" }] }, tmpDir); + await enqueueDelivery({ channel: "telegram", to: "2", payloads: [{ text: "b" }] }, tmpDir); + + const deliver = vi.fn().mockResolvedValue([]); + const log = { info: vi.fn(), warn: vi.fn(), error: vi.fn() }; + + const result = await recoverPendingDeliveries({ + deliver, + log, + cfg: baseCfg, + stateDir: tmpDir, + delay: noopDelay, + }); + + expect(deliver).toHaveBeenCalledTimes(2); + expect(result.recovered).toBe(2); + expect(result.failed).toBe(0); + expect(result.skipped).toBe(0); + + // Queue should be empty after recovery. + const remaining = await loadPendingDeliveries(tmpDir); + expect(remaining).toHaveLength(0); + }); + + it("moves entries that exceeded max retries to failed/", async () => { + // Create an entry and manually set retryCount to MAX_RETRIES. + const id = await enqueueDelivery( + { channel: "whatsapp", to: "+1", payloads: [{ text: "a" }] }, + tmpDir, + ); + const filePath = path.join(tmpDir, "delivery-queue", `${id}.json`); + const entry = JSON.parse(fs.readFileSync(filePath, "utf-8")); + entry.retryCount = MAX_RETRIES; + fs.writeFileSync(filePath, JSON.stringify(entry), "utf-8"); + + const deliver = vi.fn(); + const log = { info: vi.fn(), warn: vi.fn(), error: vi.fn() }; + + const result = await recoverPendingDeliveries({ + deliver, + log, + cfg: baseCfg, + stateDir: tmpDir, + delay: noopDelay, + }); + + expect(deliver).not.toHaveBeenCalled(); + expect(result.skipped).toBe(1); + + // Entry should be in failed/ directory. + const failedDir = path.join(tmpDir, "delivery-queue", "failed"); + expect(fs.existsSync(path.join(failedDir, `${id}.json`))).toBe(true); + }); + + it("increments retryCount on failed recovery attempt", async () => { + await enqueueDelivery({ channel: "slack", to: "#ch", payloads: [{ text: "x" }] }, tmpDir); + + const deliver = vi.fn().mockRejectedValue(new Error("network down")); + const log = { info: vi.fn(), warn: vi.fn(), error: vi.fn() }; + + const result = await recoverPendingDeliveries({ + deliver, + log, + cfg: baseCfg, + stateDir: tmpDir, + delay: noopDelay, + }); + + expect(result.failed).toBe(1); + expect(result.recovered).toBe(0); + + // Entry should still be in queue with incremented retryCount. + const entries = await loadPendingDeliveries(tmpDir); + expect(entries).toHaveLength(1); + expect(entries[0].retryCount).toBe(1); + expect(entries[0].lastError).toBe("network down"); + }); + + it("passes skipQueue: true to prevent re-enqueueing during recovery", async () => { + await enqueueDelivery({ channel: "whatsapp", to: "+1", payloads: [{ text: "a" }] }, tmpDir); + + const deliver = vi.fn().mockResolvedValue([]); + const log = { info: vi.fn(), warn: vi.fn(), error: vi.fn() }; + + await recoverPendingDeliveries({ + deliver, + log, + cfg: baseCfg, + stateDir: tmpDir, + delay: noopDelay, + }); + + expect(deliver).toHaveBeenCalledWith(expect.objectContaining({ skipQueue: true })); + }); + + it("replays stored delivery options during recovery", async () => { + await enqueueDelivery( + { + channel: "whatsapp", + to: "+1", + payloads: [{ text: "a" }], + bestEffort: true, + gifPlayback: true, + silent: true, + mirror: { + sessionKey: "agent:main:main", + text: "a", + mediaUrls: ["https://example.com/a.png"], + }, + }, + tmpDir, + ); + + const deliver = vi.fn().mockResolvedValue([]); + const log = { info: vi.fn(), warn: vi.fn(), error: vi.fn() }; + + await recoverPendingDeliveries({ + deliver, + log, + cfg: baseCfg, + stateDir: tmpDir, + delay: noopDelay, + }); + + expect(deliver).toHaveBeenCalledWith( + expect.objectContaining({ + bestEffort: true, + gifPlayback: true, + silent: true, + mirror: { + sessionKey: "agent:main:main", + text: "a", + mediaUrls: ["https://example.com/a.png"], + }, + }), + ); + }); + + it("respects maxRecoveryMs time budget", async () => { + await enqueueDelivery({ channel: "whatsapp", to: "+1", payloads: [{ text: "a" }] }, tmpDir); + await enqueueDelivery({ channel: "telegram", to: "2", payloads: [{ text: "b" }] }, tmpDir); + await enqueueDelivery({ channel: "slack", to: "#c", payloads: [{ text: "c" }] }, tmpDir); + + const deliver = vi.fn().mockResolvedValue([]); + const log = { info: vi.fn(), warn: vi.fn(), error: vi.fn() }; + + const result = await recoverPendingDeliveries({ + deliver, + log, + cfg: baseCfg, + stateDir: tmpDir, + delay: noopDelay, + maxRecoveryMs: 0, // Immediate timeout β€” no entries should be processed. + }); + + expect(deliver).not.toHaveBeenCalled(); + expect(result.recovered).toBe(0); + expect(result.failed).toBe(0); + expect(result.skipped).toBe(0); + + // All entries should still be in the queue. + const remaining = await loadPendingDeliveries(tmpDir); + expect(remaining).toHaveLength(3); + + // Should have logged a warning about deferred entries. + expect(log.warn).toHaveBeenCalledWith(expect.stringContaining("deferred to next restart")); + }); + + it("defers entries when backoff exceeds the recovery budget", async () => { + const id = await enqueueDelivery( + { channel: "whatsapp", to: "+1", payloads: [{ text: "a" }] }, + tmpDir, + ); + const filePath = path.join(tmpDir, "delivery-queue", `${id}.json`); + const entry = JSON.parse(fs.readFileSync(filePath, "utf-8")); + entry.retryCount = 3; + fs.writeFileSync(filePath, JSON.stringify(entry), "utf-8"); + + const deliver = vi.fn().mockResolvedValue([]); + const delay = vi.fn(async () => {}); + const log = { info: vi.fn(), warn: vi.fn(), error: vi.fn() }; + + const result = await recoverPendingDeliveries({ + deliver, + log, + cfg: baseCfg, + stateDir: tmpDir, + delay, + maxRecoveryMs: 1000, + }); + + expect(deliver).not.toHaveBeenCalled(); + expect(delay).not.toHaveBeenCalled(); + expect(result).toEqual({ recovered: 0, failed: 0, skipped: 0 }); + + const remaining = await loadPendingDeliveries(tmpDir); + expect(remaining).toHaveLength(1); + + expect(log.warn).toHaveBeenCalledWith(expect.stringContaining("deferred to next restart")); + }); + + it("returns zeros when queue is empty", async () => { + const deliver = vi.fn(); + const log = { info: vi.fn(), warn: vi.fn(), error: vi.fn() }; + + const result = await recoverPendingDeliveries({ + deliver, + log, + cfg: baseCfg, + stateDir: tmpDir, + delay: noopDelay, + }); + + expect(result).toEqual({ recovered: 0, failed: 0, skipped: 0 }); + expect(deliver).not.toHaveBeenCalled(); + }); +}); diff --git a/src/infra/outbound/delivery-queue.ts b/src/infra/outbound/delivery-queue.ts new file mode 100644 index 00000000000..7303d827243 --- /dev/null +++ b/src/infra/outbound/delivery-queue.ts @@ -0,0 +1,328 @@ +import crypto from "node:crypto"; +import fs from "node:fs"; +import path from "node:path"; +import type { ReplyPayload } from "../../auto-reply/types.js"; +import type { OpenClawConfig } from "../../config/config.js"; +import type { OutboundChannel } from "./targets.js"; +import { resolveStateDir } from "../../config/paths.js"; + +const QUEUE_DIRNAME = "delivery-queue"; +const FAILED_DIRNAME = "failed"; +const MAX_RETRIES = 5; + +/** Backoff delays in milliseconds indexed by retry count (1-based). */ +const BACKOFF_MS: readonly number[] = [ + 5_000, // retry 1: 5s + 25_000, // retry 2: 25s + 120_000, // retry 3: 2m + 600_000, // retry 4: 10m +]; + +export interface QueuedDelivery { + id: string; + enqueuedAt: number; + channel: Exclude; + to: string; + accountId?: string; + /** + * Original payloads before plugin hooks. On recovery, hooks re-run on these + * payloads β€” this is intentional since hooks are stateless transforms and + * should produce the same result on replay. + */ + payloads: ReplyPayload[]; + threadId?: string | number | null; + replyToId?: string | null; + bestEffort?: boolean; + gifPlayback?: boolean; + silent?: boolean; + mirror?: { + sessionKey: string; + agentId?: string; + text?: string; + mediaUrls?: string[]; + }; + retryCount: number; + lastError?: string; +} + +function resolveQueueDir(stateDir?: string): string { + const base = stateDir ?? resolveStateDir(); + return path.join(base, QUEUE_DIRNAME); +} + +function resolveFailedDir(stateDir?: string): string { + return path.join(resolveQueueDir(stateDir), FAILED_DIRNAME); +} + +/** Ensure the queue directory (and failed/ subdirectory) exist. */ +export async function ensureQueueDir(stateDir?: string): Promise { + const queueDir = resolveQueueDir(stateDir); + await fs.promises.mkdir(queueDir, { recursive: true, mode: 0o700 }); + await fs.promises.mkdir(resolveFailedDir(stateDir), { recursive: true, mode: 0o700 }); + return queueDir; +} + +/** Persist a delivery entry to disk before attempting send. Returns the entry ID. */ +export async function enqueueDelivery( + params: { + channel: Exclude; + to: string; + accountId?: string; + payloads: ReplyPayload[]; + threadId?: string | number | null; + replyToId?: string | null; + bestEffort?: boolean; + gifPlayback?: boolean; + silent?: boolean; + mirror?: { + sessionKey: string; + agentId?: string; + text?: string; + mediaUrls?: string[]; + }; + }, + stateDir?: string, +): Promise { + const queueDir = await ensureQueueDir(stateDir); + const id = crypto.randomUUID(); + const entry: QueuedDelivery = { + id, + enqueuedAt: Date.now(), + channel: params.channel, + to: params.to, + accountId: params.accountId, + payloads: params.payloads, + threadId: params.threadId, + replyToId: params.replyToId, + bestEffort: params.bestEffort, + gifPlayback: params.gifPlayback, + silent: params.silent, + mirror: params.mirror, + retryCount: 0, + }; + const filePath = path.join(queueDir, `${id}.json`); + const tmp = `${filePath}.${process.pid}.tmp`; + const json = JSON.stringify(entry, null, 2); + await fs.promises.writeFile(tmp, json, { encoding: "utf-8", mode: 0o600 }); + await fs.promises.rename(tmp, filePath); + return id; +} + +/** Remove a successfully delivered entry from the queue. */ +export async function ackDelivery(id: string, stateDir?: string): Promise { + const filePath = path.join(resolveQueueDir(stateDir), `${id}.json`); + try { + await fs.promises.unlink(filePath); + } catch (err) { + const code = + err && typeof err === "object" && "code" in err + ? String((err as { code?: unknown }).code) + : null; + if (code !== "ENOENT") { + throw err; + } + // Already removed β€” no-op. + } +} + +/** Update a queue entry after a failed delivery attempt. */ +export async function failDelivery(id: string, error: string, stateDir?: string): Promise { + const filePath = path.join(resolveQueueDir(stateDir), `${id}.json`); + const raw = await fs.promises.readFile(filePath, "utf-8"); + const entry: QueuedDelivery = JSON.parse(raw); + entry.retryCount += 1; + entry.lastError = error; + const tmp = `${filePath}.${process.pid}.tmp`; + await fs.promises.writeFile(tmp, JSON.stringify(entry, null, 2), { + encoding: "utf-8", + mode: 0o600, + }); + await fs.promises.rename(tmp, filePath); +} + +/** Load all pending delivery entries from the queue directory. */ +export async function loadPendingDeliveries(stateDir?: string): Promise { + const queueDir = resolveQueueDir(stateDir); + let files: string[]; + try { + files = await fs.promises.readdir(queueDir); + } catch (err) { + const code = + err && typeof err === "object" && "code" in err + ? String((err as { code?: unknown }).code) + : null; + if (code === "ENOENT") { + return []; + } + throw err; + } + const entries: QueuedDelivery[] = []; + for (const file of files) { + if (!file.endsWith(".json")) { + continue; + } + const filePath = path.join(queueDir, file); + try { + const stat = await fs.promises.stat(filePath); + if (!stat.isFile()) { + continue; + } + const raw = await fs.promises.readFile(filePath, "utf-8"); + entries.push(JSON.parse(raw)); + } catch { + // Skip malformed or inaccessible entries. + } + } + return entries; +} + +/** Move a queue entry to the failed/ subdirectory. */ +export async function moveToFailed(id: string, stateDir?: string): Promise { + const queueDir = resolveQueueDir(stateDir); + const failedDir = resolveFailedDir(stateDir); + await fs.promises.mkdir(failedDir, { recursive: true, mode: 0o700 }); + const src = path.join(queueDir, `${id}.json`); + const dest = path.join(failedDir, `${id}.json`); + await fs.promises.rename(src, dest); +} + +/** Compute the backoff delay in ms for a given retry count. */ +export function computeBackoffMs(retryCount: number): number { + if (retryCount <= 0) { + return 0; + } + return BACKOFF_MS[Math.min(retryCount - 1, BACKOFF_MS.length - 1)] ?? BACKOFF_MS.at(-1) ?? 0; +} + +export type DeliverFn = (params: { + cfg: OpenClawConfig; + channel: Exclude; + to: string; + accountId?: string; + payloads: ReplyPayload[]; + threadId?: string | number | null; + replyToId?: string | null; + bestEffort?: boolean; + gifPlayback?: boolean; + silent?: boolean; + mirror?: { + sessionKey: string; + agentId?: string; + text?: string; + mediaUrls?: string[]; + }; + skipQueue?: boolean; +}) => Promise; + +export interface RecoveryLogger { + info(msg: string): void; + warn(msg: string): void; + error(msg: string): void; +} + +/** + * On gateway startup, scan the delivery queue and retry any pending entries. + * Uses exponential backoff and moves entries that exceed MAX_RETRIES to failed/. + */ +export async function recoverPendingDeliveries(opts: { + deliver: DeliverFn; + log: RecoveryLogger; + cfg: OpenClawConfig; + stateDir?: string; + /** Override for testing β€” resolves instead of using real setTimeout. */ + delay?: (ms: number) => Promise; + /** Maximum wall-clock time for recovery in ms. Remaining entries are deferred to next restart. Default: 60 000. */ + maxRecoveryMs?: number; +}): Promise<{ recovered: number; failed: number; skipped: number }> { + const pending = await loadPendingDeliveries(opts.stateDir); + if (pending.length === 0) { + return { recovered: 0, failed: 0, skipped: 0 }; + } + + // Process oldest first. + pending.sort((a, b) => a.enqueuedAt - b.enqueuedAt); + + opts.log.info(`Found ${pending.length} pending delivery entries β€” starting recovery`); + + const delayFn = opts.delay ?? ((ms: number) => new Promise((r) => setTimeout(r, ms))); + const deadline = Date.now() + (opts.maxRecoveryMs ?? 60_000); + + let recovered = 0; + let failed = 0; + let skipped = 0; + + for (const entry of pending) { + const now = Date.now(); + if (now >= deadline) { + const deferred = pending.length - recovered - failed - skipped; + opts.log.warn(`Recovery time budget exceeded β€” ${deferred} entries deferred to next restart`); + break; + } + if (entry.retryCount >= MAX_RETRIES) { + opts.log.warn( + `Delivery ${entry.id} exceeded max retries (${entry.retryCount}/${MAX_RETRIES}) β€” moving to failed/`, + ); + try { + await moveToFailed(entry.id, opts.stateDir); + } catch (err) { + opts.log.error(`Failed to move entry ${entry.id} to failed/: ${String(err)}`); + } + skipped += 1; + continue; + } + + const backoff = computeBackoffMs(entry.retryCount + 1); + if (backoff > 0) { + if (now + backoff >= deadline) { + const deferred = pending.length - recovered - failed - skipped; + opts.log.warn( + `Recovery time budget exceeded β€” ${deferred} entries deferred to next restart`, + ); + break; + } + opts.log.info(`Waiting ${backoff}ms before retrying delivery ${entry.id}`); + await delayFn(backoff); + } + + try { + await opts.deliver({ + cfg: opts.cfg, + channel: entry.channel, + to: entry.to, + accountId: entry.accountId, + payloads: entry.payloads, + threadId: entry.threadId, + replyToId: entry.replyToId, + bestEffort: entry.bestEffort, + gifPlayback: entry.gifPlayback, + silent: entry.silent, + mirror: entry.mirror, + skipQueue: true, // Prevent re-enqueueing during recovery + }); + await ackDelivery(entry.id, opts.stateDir); + recovered += 1; + opts.log.info(`Recovered delivery ${entry.id} to ${entry.channel}:${entry.to}`); + } catch (err) { + try { + await failDelivery( + entry.id, + err instanceof Error ? err.message : String(err), + opts.stateDir, + ); + } catch { + // Best-effort update. + } + failed += 1; + opts.log.warn( + `Retry failed for delivery ${entry.id}: ${err instanceof Error ? err.message : String(err)}`, + ); + } + } + + opts.log.info( + `Delivery recovery complete: ${recovered} recovered, ${failed} failed, ${skipped} skipped (max retries)`, + ); + return { recovered, failed, skipped }; +} + +export { MAX_RETRIES }; diff --git a/src/infra/ports.ts b/src/infra/ports.ts index f8bc799c578..1d73b7ff64e 100644 --- a/src/infra/ports.ts +++ b/src/infra/ports.ts @@ -42,8 +42,7 @@ export async function ensurePortAvailable(port: number): Promise { }); } catch (err) { if (isErrno(err) && err.code === "EADDRINUSE") { - const details = await describePortOwner(port); - throw new PortInUseError(port, details); + throw new PortInUseError(port); } throw err; } @@ -57,7 +56,10 @@ export async function handlePortError( ): Promise { // Uniform messaging for EADDRINUSE with optional owner details. if (err instanceof PortInUseError || (isErrno(err) && err.code === "EADDRINUSE")) { - const details = err instanceof PortInUseError ? err.details : await describePortOwner(port); + const details = + err instanceof PortInUseError + ? (err.details ?? (await describePortOwner(port))) + : await describePortOwner(port); runtime.error(danger(`${context} failed: port ${port} is already in use.`)); if (details) { runtime.error(info("Port listener details:")); diff --git a/src/infra/restart-sentinel.test.ts b/src/infra/restart-sentinel.test.ts index 638d389f561..5c1fa60632b 100644 --- a/src/infra/restart-sentinel.test.ts +++ b/src/infra/restart-sentinel.test.ts @@ -4,6 +4,7 @@ import path from "node:path"; import { afterEach, beforeEach, describe, expect, it } from "vitest"; import { consumeRestartSentinel, + formatRestartSentinelMessage, readRestartSentinel, resolveRestartSentinelPath, trimLogTail, @@ -61,6 +62,40 @@ describe("restart sentinel", () => { await expect(fs.stat(filePath)).rejects.toThrow(); }); + it("formatRestartSentinelMessage uses custom message when present", () => { + const payload = { + kind: "config-apply" as const, + status: "ok" as const, + ts: Date.now(), + message: "Config updated successfully", + }; + expect(formatRestartSentinelMessage(payload)).toBe("Config updated successfully"); + }); + + it("formatRestartSentinelMessage falls back to summary when no message", () => { + const payload = { + kind: "update" as const, + status: "ok" as const, + ts: Date.now(), + stats: { mode: "git" }, + }; + const result = formatRestartSentinelMessage(payload); + expect(result).toContain("Gateway restart"); + expect(result).toContain("update"); + expect(result).toContain("ok"); + }); + + it("formatRestartSentinelMessage falls back to summary for blank message", () => { + const payload = { + kind: "restart" as const, + status: "ok" as const, + ts: Date.now(), + message: " ", + }; + const result = formatRestartSentinelMessage(payload); + expect(result).toContain("Gateway restart"); + }); + it("trims log tails", () => { const text = "a".repeat(9000); const trimmed = trimLogTail(text, 8000); diff --git a/src/infra/restart-sentinel.ts b/src/infra/restart-sentinel.ts index 1f3b13094f9..8405426cbd6 100644 --- a/src/infra/restart-sentinel.ts +++ b/src/infra/restart-sentinel.ts @@ -28,7 +28,7 @@ export type RestartSentinelStats = { }; export type RestartSentinelPayload = { - kind: "config-apply" | "update" | "restart"; + kind: "config-apply" | "config-patch" | "update" | "restart"; status: "ok" | "error" | "skipped"; ts: number; sessionKey?: string; @@ -109,7 +109,10 @@ export async function consumeRestartSentinel( } export function formatRestartSentinelMessage(payload: RestartSentinelPayload): string { - return `GatewayRestart:\n${JSON.stringify(payload, null, 2)}`; + if (payload.message?.trim()) { + return payload.message.trim(); + } + return summarizeRestartSentinel(payload); } export function summarizeRestartSentinel(payload: RestartSentinelPayload): string { diff --git a/src/infra/restart.ts b/src/infra/restart.ts index d671c112b53..60540884b90 100644 --- a/src/infra/restart.ts +++ b/src/infra/restart.ts @@ -13,10 +13,55 @@ export type RestartAttempt = { const SPAWN_TIMEOUT_MS = 2000; const SIGUSR1_AUTH_GRACE_MS = 5000; +const DEFAULT_DEFERRAL_POLL_MS = 500; +const DEFAULT_DEFERRAL_MAX_WAIT_MS = 30_000; let sigusr1AuthorizedCount = 0; let sigusr1AuthorizedUntil = 0; let sigusr1ExternalAllowed = false; +let preRestartCheck: (() => number) | null = null; +let restartCycleToken = 0; +let emittedRestartToken = 0; +let consumedRestartToken = 0; + +function hasUnconsumedRestartSignal(): boolean { + return emittedRestartToken > consumedRestartToken; +} + +/** + * Register a callback that scheduleGatewaySigusr1Restart checks before emitting SIGUSR1. + * The callback should return the number of pending items (0 = safe to restart). + */ +export function setPreRestartDeferralCheck(fn: () => number): void { + preRestartCheck = fn; +} + +/** + * Emit an authorized SIGUSR1 gateway restart, guarded against duplicate emissions. + * Returns true if SIGUSR1 was emitted, false if a restart was already emitted. + * Both scheduleGatewaySigusr1Restart and the config watcher should use this + * to ensure only one restart fires. + */ +export function emitGatewayRestart(): boolean { + if (hasUnconsumedRestartSignal()) { + return false; + } + const cycleToken = ++restartCycleToken; + emittedRestartToken = cycleToken; + authorizeGatewaySigusr1Restart(); + try { + if (process.listenerCount("SIGUSR1") > 0) { + process.emit("SIGUSR1"); + } else { + process.kill(process.pid, "SIGUSR1"); + } + } catch { + // Roll back the cycle marker so future restart requests can still proceed. + emittedRestartToken = consumedRestartToken; + return false; + } + return true; +} function resetSigusr1AuthorizationIfExpired(now = Date.now()) { if (sigusr1AuthorizedCount <= 0) { @@ -37,7 +82,7 @@ export function isGatewaySigusr1RestartExternallyAllowed() { return sigusr1ExternalAllowed; } -export function authorizeGatewaySigusr1Restart(delayMs = 0) { +function authorizeGatewaySigusr1Restart(delayMs = 0) { const delay = Math.max(0, Math.floor(delayMs)); const expiresAt = Date.now() + delay + SIGUSR1_AUTH_GRACE_MS; sigusr1AuthorizedCount += 1; @@ -58,6 +103,80 @@ export function consumeGatewaySigusr1RestartAuthorization(): boolean { return true; } +/** + * Mark the currently emitted SIGUSR1 restart cycle as consumed by the run loop. + * This explicitly advances the cycle state instead of resetting emit guards inside + * consumeGatewaySigusr1RestartAuthorization(). + */ +export function markGatewaySigusr1RestartHandled(): void { + if (hasUnconsumedRestartSignal()) { + consumedRestartToken = emittedRestartToken; + } +} + +export type RestartDeferralHooks = { + onDeferring?: (pending: number) => void; + onReady?: () => void; + onTimeout?: (pending: number, elapsedMs: number) => void; + onCheckError?: (err: unknown) => void; +}; + +/** + * Poll pending work until it drains (or times out), then emit one restart signal. + * Shared by both the direct RPC restart path and the config watcher path. + */ +export function deferGatewayRestartUntilIdle(opts: { + getPendingCount: () => number; + hooks?: RestartDeferralHooks; + pollMs?: number; + maxWaitMs?: number; +}): void { + const pollMsRaw = opts.pollMs ?? DEFAULT_DEFERRAL_POLL_MS; + const pollMs = Math.max(10, Math.floor(pollMsRaw)); + const maxWaitMsRaw = opts.maxWaitMs ?? DEFAULT_DEFERRAL_MAX_WAIT_MS; + const maxWaitMs = Math.max(pollMs, Math.floor(maxWaitMsRaw)); + + let pending: number; + try { + pending = opts.getPendingCount(); + } catch (err) { + opts.hooks?.onCheckError?.(err); + emitGatewayRestart(); + return; + } + if (pending <= 0) { + opts.hooks?.onReady?.(); + emitGatewayRestart(); + return; + } + + opts.hooks?.onDeferring?.(pending); + const startedAt = Date.now(); + const poll = setInterval(() => { + let current: number; + try { + current = opts.getPendingCount(); + } catch (err) { + clearInterval(poll); + opts.hooks?.onCheckError?.(err); + emitGatewayRestart(); + return; + } + if (current <= 0) { + clearInterval(poll); + opts.hooks?.onReady?.(); + emitGatewayRestart(); + return; + } + const elapsedMs = Date.now() - startedAt; + if (elapsedMs >= maxWaitMs) { + clearInterval(poll); + opts.hooks?.onTimeout?.(current, elapsedMs); + emitGatewayRestart(); + } + }, pollMs); +} + function formatSpawnDetail(result: { error?: unknown; status?: number | null; @@ -189,27 +308,22 @@ export function scheduleGatewaySigusr1Restart(opts?: { typeof opts?.reason === "string" && opts.reason.trim() ? opts.reason.trim().slice(0, 200) : undefined; - authorizeGatewaySigusr1Restart(delayMs); - const pid = process.pid; - const hasListener = process.listenerCount("SIGUSR1") > 0; + setTimeout(() => { - try { - if (hasListener) { - process.emit("SIGUSR1"); - } else { - process.kill(pid, "SIGUSR1"); - } - } catch { - /* ignore */ + const pendingCheck = preRestartCheck; + if (!pendingCheck) { + emitGatewayRestart(); + return; } + deferGatewayRestartUntilIdle({ getPendingCount: pendingCheck }); }, delayMs); return { ok: true, - pid, + pid: process.pid, signal: "SIGUSR1", delayMs, reason, - mode: hasListener ? "emit" : "signal", + mode: process.listenerCount("SIGUSR1") > 0 ? "emit" : "signal", }; } @@ -218,5 +332,9 @@ export const __testing = { sigusr1AuthorizedCount = 0; sigusr1AuthorizedUntil = 0; sigusr1ExternalAllowed = false; + preRestartCheck = null; + restartCycleToken = 0; + emittedRestartToken = 0; + consumedRestartToken = 0; }, }; diff --git a/src/infra/transport-ready.test.ts b/src/infra/transport-ready.test.ts index adb2560ce16..2df90a6420e 100644 --- a/src/infra/transport-ready.test.ts +++ b/src/infra/transport-ready.test.ts @@ -15,22 +15,22 @@ describe("waitForTransportReady", () => { let attempts = 0; const readyPromise = waitForTransportReady({ label: "test transport", - timeoutMs: 500, - logAfterMs: 120, - logIntervalMs: 100, - pollIntervalMs: 80, + timeoutMs: 220, + logAfterMs: 60, + logIntervalMs: 1_000, + pollIntervalMs: 50, runtime, check: async () => { attempts += 1; - if (attempts > 4) { + if (attempts > 2) { return { ok: true }; } return { ok: false, error: "not ready" }; }, }); - for (let i = 0; i < 5; i += 1) { - await vi.advanceTimersByTimeAsync(80); + for (let i = 0; i < 3; i += 1) { + await vi.advanceTimersByTimeAsync(50); } await readyPromise; @@ -41,14 +41,14 @@ describe("waitForTransportReady", () => { const runtime = { log: vi.fn(), error: vi.fn(), exit: vi.fn() }; const waitPromise = waitForTransportReady({ label: "test transport", - timeoutMs: 200, + timeoutMs: 110, logAfterMs: 0, - logIntervalMs: 100, + logIntervalMs: 1_000, pollIntervalMs: 50, runtime, check: async () => ({ ok: false, error: "still down" }), }); - await vi.advanceTimersByTimeAsync(250); + await vi.advanceTimersByTimeAsync(200); await expect(waitPromise).rejects.toThrow("test transport not ready"); expect(runtime.error).toHaveBeenCalled(); }); diff --git a/src/logger.test.ts b/src/logger.test.ts index 9f87d4b3794..93523906e3f 100644 --- a/src/logger.test.ts +++ b/src/logger.test.ts @@ -67,7 +67,7 @@ describe("logger helpers", () => { it("uses daily rolling default log file and prunes old ones", () => { resetLogger(); - setLoggerOverride({}); // force defaults regardless of user config + setLoggerOverride({ level: "info" }); // force default file path with enabled file logging const today = localDateString(new Date()); const todayPath = path.join(DEFAULT_LOG_DIR, `openclaw-${today}.log`); diff --git a/src/logging/console.ts b/src/logging/console.ts index ad3d99a2efd..879a250676c 100644 --- a/src/logging/console.ts +++ b/src/logging/console.ts @@ -37,6 +37,9 @@ function normalizeConsoleLevel(level?: string): LogLevel { if (isVerbose()) { return "debug"; } + if (!level && process.env.VITEST === "true" && process.env.OPENCLAW_TEST_CONSOLE !== "1") { + return "silent"; + } return normalizeLogLevel(level, "info"); } diff --git a/src/logging/logger.ts b/src/logging/logger.ts index 63de56aed21..b3ddd65d92e 100644 --- a/src/logging/logger.ts +++ b/src/logging/logger.ts @@ -63,7 +63,9 @@ function resolveSettings(): ResolvedSettings { cfg = undefined; } } - const level = normalizeLogLevel(cfg?.level, "info"); + const defaultLevel = + process.env.VITEST === "true" && process.env.OPENCLAW_TEST_FILE_LOG !== "1" ? "silent" : "info"; + const level = normalizeLogLevel(cfg?.level, defaultLevel); const file = cfg?.file ?? defaultRollingPathForToday(); return { level, file }; } diff --git a/src/logging/subsystem.test.ts b/src/logging/subsystem.test.ts new file mode 100644 index 00000000000..e389d78ba8a --- /dev/null +++ b/src/logging/subsystem.test.ts @@ -0,0 +1,56 @@ +import { afterEach, describe, expect, it } from "vitest"; +import { setConsoleSubsystemFilter } from "./console.js"; +import { resetLogger, setLoggerOverride } from "./logger.js"; +import { createSubsystemLogger } from "./subsystem.js"; + +afterEach(() => { + setConsoleSubsystemFilter(null); + setLoggerOverride(null); + resetLogger(); +}); + +describe("createSubsystemLogger().isEnabled", () => { + it("returns true for any/file when only file logging would emit", () => { + setLoggerOverride({ level: "debug", consoleLevel: "silent" }); + const log = createSubsystemLogger("agent/embedded"); + + expect(log.isEnabled("debug")).toBe(true); + expect(log.isEnabled("debug", "file")).toBe(true); + expect(log.isEnabled("debug", "console")).toBe(false); + }); + + it("returns true for any/console when only console logging would emit", () => { + setLoggerOverride({ level: "silent", consoleLevel: "debug" }); + const log = createSubsystemLogger("agent/embedded"); + + expect(log.isEnabled("debug")).toBe(true); + expect(log.isEnabled("debug", "console")).toBe(true); + expect(log.isEnabled("debug", "file")).toBe(false); + }); + + it("returns false when neither console nor file logging would emit", () => { + setLoggerOverride({ level: "silent", consoleLevel: "silent" }); + const log = createSubsystemLogger("agent/embedded"); + + expect(log.isEnabled("debug")).toBe(false); + expect(log.isEnabled("debug", "console")).toBe(false); + expect(log.isEnabled("debug", "file")).toBe(false); + }); + + it("honors console subsystem filters for console target", () => { + setLoggerOverride({ level: "silent", consoleLevel: "info" }); + setConsoleSubsystemFilter(["gateway"]); + const log = createSubsystemLogger("agent/embedded"); + + expect(log.isEnabled("info", "console")).toBe(false); + }); + + it("does not apply console subsystem filters to file target", () => { + setLoggerOverride({ level: "info", consoleLevel: "silent" }); + setConsoleSubsystemFilter(["gateway"]); + const log = createSubsystemLogger("agent/embedded"); + + expect(log.isEnabled("info", "file")).toBe(true); + expect(log.isEnabled("info")).toBe(true); + }); +}); diff --git a/src/logging/subsystem.ts b/src/logging/subsystem.ts index a1ec00abc29..89671b7b90a 100644 --- a/src/logging/subsystem.ts +++ b/src/logging/subsystem.ts @@ -6,13 +6,14 @@ import { defaultRuntime, type RuntimeEnv } from "../runtime.js"; import { clearActiveProgressLine } from "../terminal/progress-line.js"; import { getConsoleSettings, shouldLogSubsystemToConsole } from "./console.js"; import { type LogLevel, levelToMinLevel } from "./levels.js"; -import { getChildLogger } from "./logger.js"; +import { getChildLogger, isFileLogLevelEnabled } from "./logger.js"; import { loggingState } from "./state.js"; type LogObj = { date?: Date } & Record; export type SubsystemLogger = { subsystem: string; + isEnabled: (level: LogLevel, target?: "any" | "console" | "file") => boolean; trace: (message: string, meta?: Record) => void; debug: (message: string, meta?: Record) => void; info: (message: string, meta?: Record) => void; @@ -271,9 +272,26 @@ export function createSubsystemLogger(subsystem: string): SubsystemLogger { }); writeConsoleLine(level, line); }; + const isConsoleEnabled = (level: LogLevel): boolean => { + const consoleSettings = getConsoleSettings(); + return ( + shouldLogToConsole(level, { level: consoleSettings.level }) && + shouldLogSubsystemToConsole(subsystem) + ); + }; + const isFileEnabled = (level: LogLevel): boolean => isFileLogLevelEnabled(level); const logger: SubsystemLogger = { subsystem, + isEnabled: (level, target = "any") => { + if (target === "console") { + return isConsoleEnabled(level); + } + if (target === "file") { + return isFileEnabled(level); + } + return isConsoleEnabled(level) || isFileEnabled(level); + }, trace: (message, meta) => emit("trace", message, meta), debug: (message, meta) => emit("debug", message, meta), info: (message, meta) => emit("info", message, meta), diff --git a/src/macos/gateway-daemon.ts b/src/macos/gateway-daemon.ts index eb02c060640..a33ca94e81c 100644 --- a/src/macos/gateway-daemon.ts +++ b/src/macos/gateway-daemon.ts @@ -49,9 +49,15 @@ async function main() { { setGatewayWsLogStyle }, { setVerbose }, { acquireGatewayLock, GatewayLockError }, - { consumeGatewaySigusr1RestartAuthorization, isGatewaySigusr1RestartExternallyAllowed }, + { + consumeGatewaySigusr1RestartAuthorization, + isGatewaySigusr1RestartExternallyAllowed, + markGatewaySigusr1RestartHandled, + }, { defaultRuntime }, { enableConsoleCapture, setConsoleTimestampPrefix }, + commandQueueMod, + { createRestartIterationHook }, ] = await Promise.all([ import("../config/config.js"), import("../gateway/server.js"), @@ -61,6 +67,8 @@ async function main() { import("../infra/restart.js"), import("../runtime.js"), import("../logging.js"), + import("../process/command-queue.js"), + import("../process/restart-recovery.js"), ] as const); enableConsoleCapture(); @@ -132,14 +140,32 @@ async function main() { `gateway: received ${signal}; ${isRestart ? "restarting" : "shutting down"}`, ); + const DRAIN_TIMEOUT_MS = 30_000; + const SHUTDOWN_TIMEOUT_MS = 5_000; + const forceExitMs = isRestart ? DRAIN_TIMEOUT_MS + SHUTDOWN_TIMEOUT_MS : SHUTDOWN_TIMEOUT_MS; forceExitTimer = setTimeout(() => { defaultRuntime.error("gateway: shutdown timed out; exiting without full cleanup"); cleanupSignals(); process.exit(0); - }, 5000); + }, forceExitMs); void (async () => { try { + if (isRestart) { + const activeTasks = commandQueueMod.getActiveTaskCount(); + if (activeTasks > 0) { + defaultRuntime.log( + `gateway: draining ${activeTasks} active task(s) before restart (timeout ${DRAIN_TIMEOUT_MS}ms)`, + ); + const { drained } = await commandQueueMod.waitForActiveTasks(DRAIN_TIMEOUT_MS); + if (drained) { + defaultRuntime.log("gateway: all active tasks drained"); + } else { + defaultRuntime.log("gateway: drain timeout reached; proceeding with restart"); + } + } + } + await server?.close({ reason: isRestart ? "gateway restarting" : "gateway stopping", restartExpectedMs: isRestart ? 1500 : null, @@ -179,6 +205,7 @@ async function main() { ); return; } + markGatewaySigusr1RestartHandled(); request("restart", "SIGUSR1"); }; @@ -196,8 +223,17 @@ async function main() { } throw err; } + const onIteration = createRestartIterationHook(() => { + // After an in-process restart (SIGUSR1), reset command-queue lane state. + // Interrupted tasks from the previous lifecycle may have left `active` + // counts elevated (their finally blocks never ran), permanently blocking + // new work from draining. + commandQueueMod.resetAllLanes(); + }); + // eslint-disable-next-line no-constant-condition while (true) { + onIteration(); try { server = await startGatewayServer(port, { bind }); } catch (err) { @@ -210,7 +246,7 @@ async function main() { }); } } finally { - await (lock as GatewayLockHandle | null)?.release(); + await lock?.release(); cleanupSignals(); } } diff --git a/src/media/constants.test.ts b/src/media/constants.test.ts new file mode 100644 index 00000000000..613f4c4b381 --- /dev/null +++ b/src/media/constants.test.ts @@ -0,0 +1,14 @@ +import { describe, expect, it } from "vitest"; +import { mediaKindFromMime } from "./constants.js"; + +describe("mediaKindFromMime", () => { + it("classifies text mimes as document", () => { + expect(mediaKindFromMime("text/plain")).toBe("document"); + expect(mediaKindFromMime("text/csv")).toBe("document"); + expect(mediaKindFromMime("text/html; charset=utf-8")).toBe("document"); + }); + + it("keeps unknown mimes as unknown", () => { + expect(mediaKindFromMime("model/gltf+json")).toBe("unknown"); + }); +}); diff --git a/src/media/constants.ts b/src/media/constants.ts index 63fdc03fcc2..5dec8cedbfd 100644 --- a/src/media/constants.ts +++ b/src/media/constants.ts @@ -21,6 +21,9 @@ export function mediaKindFromMime(mime?: string | null): MediaKind { if (mime === "application/pdf") { return "document"; } + if (mime.startsWith("text/")) { + return "document"; + } if (mime.startsWith("application/")) { return "document"; } diff --git a/src/memory/index.test.ts b/src/memory/index.test.ts index 3f01ab85593..9f5d708a2b4 100644 --- a/src/memory/index.test.ts +++ b/src/memory/index.test.ts @@ -1,7 +1,7 @@ import fs from "node:fs/promises"; import os from "node:os"; import path from "node:path"; -import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; import { getMemorySearchManager, type MemoryIndexManager } from "./index.js"; let embedBatchCalls = 0; @@ -34,21 +34,31 @@ vi.mock("./embeddings.js", () => { }); describe("memory index", () => { + let fixtureRoot = ""; + let fixtureCount = 0; let workspaceDir: string; let indexPath: string; let manager: MemoryIndexManager | null = null; + beforeAll(async () => { + fixtureRoot = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-mem-fixtures-")); + }); + + afterAll(async () => { + await fs.rm(fixtureRoot, { recursive: true, force: true }); + }); + beforeEach(async () => { embedBatchCalls = 0; failEmbeddings = false; - workspaceDir = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-mem-")); + workspaceDir = path.join(fixtureRoot, `case-${fixtureCount++}`); + await fs.mkdir(workspaceDir, { recursive: true }); indexPath = path.join(workspaceDir, "index.sqlite"); await fs.mkdir(path.join(workspaceDir, "memory")); await fs.writeFile( path.join(workspaceDir, "memory", "2026-01-12.md"), - "# Log\nAlpha memory line.\nZebra memory line.\nAnother line.", + "# Log\nAlpha memory line.\nZebra memory line.", ); - await fs.writeFile(path.join(workspaceDir, "MEMORY.md"), "Beta knowledge base entry."); }); afterEach(async () => { @@ -56,7 +66,6 @@ describe("memory index", () => { await manager.close(); manager = null; } - await fs.rm(workspaceDir, { recursive: true, force: true }); }); it("indexes memory files and searches by vector", async () => { @@ -134,6 +143,7 @@ describe("memory index", () => { throw new Error("manager missing"); } await first.manager.sync({ force: true }); + const callsAfterFirstSync = embedBatchCalls; await first.manager.close(); const second = await getMemorySearchManager({ @@ -158,8 +168,9 @@ describe("memory index", () => { } manager = second.manager; await second.manager.sync({ reason: "test" }); - const results = await second.manager.search("alpha"); - expect(results.length).toBeGreaterThan(0); + expect(embedBatchCalls).toBeGreaterThan(callsAfterFirstSync); + const status = second.manager.status(); + expect(status.files).toBeGreaterThan(0); }); it("reuses cached embeddings on forced reindex", async () => { @@ -269,8 +280,8 @@ describe("memory index", () => { expect(results[0]?.path).toContain("memory/2026-01-12.md"); }); - it("hybrid weights can favor vector-only matches over keyword-only matches", async () => { - const manyAlpha = Array.from({ length: 200 }, () => "Alpha").join(" "); + it("hybrid weights shift ranking between vector and keyword matches", async () => { + const manyAlpha = Array.from({ length: 50 }, () => "Alpha").join(" "); await fs.writeFile( path.join(workspaceDir, "memory", "vector-only.md"), "Alpha beta. Alpha beta. Alpha beta. Alpha beta.", @@ -280,7 +291,7 @@ describe("memory index", () => { `${manyAlpha} beta id123.`, ); - const cfg = { + const vectorWeightedCfg = { agents: { defaults: { workspace: workspaceDir, @@ -304,12 +315,15 @@ describe("memory index", () => { list: [{ id: "main", default: true }], }, }; - const result = await getMemorySearchManager({ cfg, agentId: "main" }); - expect(result.manager).not.toBeNull(); - if (!result.manager) { + const vectorWeighted = await getMemorySearchManager({ + cfg: vectorWeightedCfg, + agentId: "main", + }); + expect(vectorWeighted.manager).not.toBeNull(); + if (!vectorWeighted.manager) { throw new Error("manager missing"); } - manager = result.manager; + manager = vectorWeighted.manager; const status = manager.status(); if (!status.fts?.available) { @@ -317,28 +331,19 @@ describe("memory index", () => { } await manager.sync({ force: true }); - const results = await manager.search("alpha beta id123"); - expect(results.length).toBeGreaterThan(0); - const paths = results.map((r) => r.path); - expect(paths).toContain("memory/vector-only.md"); - expect(paths).toContain("memory/keyword-only.md"); - const vectorOnly = results.find((r) => r.path === "memory/vector-only.md"); - const keywordOnly = results.find((r) => r.path === "memory/keyword-only.md"); + const vectorResults = await manager.search("alpha beta id123"); + expect(vectorResults.length).toBeGreaterThan(0); + const vectorPaths = vectorResults.map((r) => r.path); + expect(vectorPaths).toContain("memory/vector-only.md"); + expect(vectorPaths).toContain("memory/keyword-only.md"); + const vectorOnly = vectorResults.find((r) => r.path === "memory/vector-only.md"); + const keywordOnly = vectorResults.find((r) => r.path === "memory/keyword-only.md"); expect((vectorOnly?.score ?? 0) > (keywordOnly?.score ?? 0)).toBe(true); - }); - it("hybrid weights can favor keyword matches when text weight dominates", async () => { - const manyAlpha = Array.from({ length: 200 }, () => "Alpha").join(" "); - await fs.writeFile( - path.join(workspaceDir, "memory", "vector-only.md"), - "Alpha beta. Alpha beta. Alpha beta. Alpha beta.", - ); - await fs.writeFile( - path.join(workspaceDir, "memory", "keyword-only.md"), - `${manyAlpha} beta id123.`, - ); + await manager.close(); + manager = null; - const cfg = { + const textWeightedCfg = { agents: { defaults: { workspace: workspaceDir, @@ -346,7 +351,7 @@ describe("memory index", () => { provider: "openai", model: "mock-embed", store: { path: indexPath, vector: { enabled: false } }, - sync: { watch: false, onSessionStart: false, onSearch: true }, + sync: { watch: false, onSessionStart: false, onSearch: false }, query: { minScore: 0, maxResults: 200, @@ -362,27 +367,21 @@ describe("memory index", () => { list: [{ id: "main", default: true }], }, }; - const result = await getMemorySearchManager({ cfg, agentId: "main" }); - expect(result.manager).not.toBeNull(); - if (!result.manager) { + + const textWeighted = await getMemorySearchManager({ cfg: textWeightedCfg, agentId: "main" }); + expect(textWeighted.manager).not.toBeNull(); + if (!textWeighted.manager) { throw new Error("manager missing"); } - manager = result.manager; - - const status = manager.status(); - if (!status.fts?.available) { - return; - } - - await manager.sync({ force: true }); - const results = await manager.search("alpha beta id123"); - expect(results.length).toBeGreaterThan(0); - const paths = results.map((r) => r.path); - expect(paths).toContain("memory/vector-only.md"); - expect(paths).toContain("memory/keyword-only.md"); - const vectorOnly = results.find((r) => r.path === "memory/vector-only.md"); - const keywordOnly = results.find((r) => r.path === "memory/keyword-only.md"); - expect((keywordOnly?.score ?? 0) > (vectorOnly?.score ?? 0)).toBe(true); + manager = textWeighted.manager; + const keywordResults = await manager.search("alpha beta id123"); + expect(keywordResults.length).toBeGreaterThan(0); + const keywordPaths = keywordResults.map((r) => r.path); + expect(keywordPaths).toContain("memory/vector-only.md"); + expect(keywordPaths).toContain("memory/keyword-only.md"); + const vectorOnlyAfter = keywordResults.find((r) => r.path === "memory/vector-only.md"); + const keywordOnlyAfter = keywordResults.find((r) => r.path === "memory/keyword-only.md"); + expect((keywordOnlyAfter?.score ?? 0) > (vectorOnlyAfter?.score ?? 0)).toBe(true); }); it("reports vector availability after probe", async () => { diff --git a/src/memory/manager.batch.test.ts b/src/memory/manager.batch.test.ts index 60586d2ec58..2cf1b30c056 100644 --- a/src/memory/manager.batch.test.ts +++ b/src/memory/manager.batch.test.ts @@ -1,7 +1,7 @@ import fs from "node:fs/promises"; import os from "node:os"; import path from "node:path"; -import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; import { getMemorySearchManager, type MemoryIndexManager } from "./index.js"; const embedBatch = vi.fn(async () => []); @@ -25,11 +25,21 @@ vi.mock("./embeddings.js", () => ({ })); describe("memory indexing with OpenAI batches", () => { + let fixtureRoot: string; + let caseId = 0; let workspaceDir: string; let indexPath: string; let manager: MemoryIndexManager | null = null; let setTimeoutSpy: ReturnType; + beforeAll(async () => { + fixtureRoot = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-mem-batch-")); + }); + + afterAll(async () => { + await fs.rm(fixtureRoot, { recursive: true, force: true }); + }); + beforeEach(async () => { embedBatch.mockClear(); embedQuery.mockClear(); @@ -48,9 +58,9 @@ describe("memory indexing with OpenAI batches", () => { } return realSetTimeout(handler, delay, ...args); }) as typeof setTimeout); - workspaceDir = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-mem-batch-")); + workspaceDir = path.join(fixtureRoot, `case-${++caseId}`); indexPath = path.join(workspaceDir, "index.sqlite"); - await fs.mkdir(path.join(workspaceDir, "memory")); + await fs.mkdir(path.join(workspaceDir, "memory"), { recursive: true }); }); afterEach(async () => { @@ -60,7 +70,6 @@ describe("memory indexing with OpenAI batches", () => { await manager.close(); manager = null; } - await fs.rm(workspaceDir, { recursive: true, force: true }); }); it("uses OpenAI batch uploads when enabled", async () => { @@ -272,7 +281,7 @@ describe("memory indexing with OpenAI batches", () => { expect(batchCreates).toBe(2); }); - it("falls back to non-batch on failure and resets failures after success", async () => { + it("tracks batch failures, resets on success, and disables after repeated failures", async () => { const content = ["flaky", "batch"].join("\n\n"); await fs.writeFile(path.join(workspaceDir, "memory", "2026-01-09.md"), content); @@ -367,12 +376,14 @@ describe("memory indexing with OpenAI batches", () => { } manager = result.manager; + // First failure: fallback to regular embeddings and increment failure count. await manager.sync({ force: true }); expect(embedBatch).toHaveBeenCalled(); let status = manager.status(); expect(status.batch?.enabled).toBe(true); expect(status.batch?.failures).toBe(1); + // Success should reset failure count. embedBatch.mockClear(); mode = "ok"; await fs.writeFile( @@ -384,110 +395,33 @@ describe("memory indexing with OpenAI batches", () => { expect(status.batch?.enabled).toBe(true); expect(status.batch?.failures).toBe(0); expect(embedBatch).not.toHaveBeenCalled(); - }); - - it("disables batch after repeated failures and skips batch thereafter", async () => { - const content = ["repeat", "failures"].join("\n\n"); - await fs.writeFile(path.join(workspaceDir, "memory", "2026-01-10.md"), content); - - let uploadedRequests: Array<{ custom_id?: string }> = []; - const fetchMock = vi.fn(async (input: RequestInfo | URL, init?: RequestInit) => { - const url = - typeof input === "string" ? input : input instanceof URL ? input.toString() : input.url; - if (url.endsWith("/files")) { - const body = init?.body; - if (!(body instanceof FormData)) { - throw new Error("expected FormData upload"); - } - for (const [key, value] of body.entries()) { - if (key !== "file") { - continue; - } - if (typeof value === "string") { - uploadedRequests = value - .split("\n") - .filter(Boolean) - .map((line) => JSON.parse(line) as { custom_id?: string }); - } else { - const text = await value.text(); - uploadedRequests = text - .split("\n") - .filter(Boolean) - .map((line) => JSON.parse(line) as { custom_id?: string }); - } - } - return new Response(JSON.stringify({ id: "file_1" }), { - status: 200, - headers: { "Content-Type": "application/json" }, - }); - } - if (url.endsWith("/batches")) { - return new Response("batch failed", { status: 500 }); - } - if (url.endsWith("/files/file_out/content")) { - const lines = uploadedRequests.map((request, index) => - JSON.stringify({ - custom_id: request.custom_id, - response: { - status_code: 200, - body: { data: [{ embedding: [index + 1, 0, 0], index: 0 }] }, - }, - }), - ); - return new Response(lines.join("\n"), { - status: 200, - headers: { "Content-Type": "application/jsonl" }, - }); - } - throw new Error(`unexpected fetch ${url}`); - }); - - vi.stubGlobal("fetch", fetchMock); - - const cfg = { - agents: { - defaults: { - workspace: workspaceDir, - memorySearch: { - provider: "openai", - model: "text-embedding-3-small", - store: { path: indexPath }, - sync: { watch: false, onSessionStart: false, onSearch: false }, - query: { minScore: 0 }, - remote: { batch: { enabled: true, wait: true, pollIntervalMs: 1 } }, - }, - }, - list: [{ id: "main", default: true }], - }, - }; - - const result = await getMemorySearchManager({ cfg, agentId: "main" }); - expect(result.manager).not.toBeNull(); - if (!result.manager) { - throw new Error("manager missing"); - } - manager = result.manager; + // Two more failures after reset should disable remote batching. + mode = "fail"; + await fs.writeFile( + path.join(workspaceDir, "memory", "2026-01-09.md"), + ["flaky", "batch", "fail-a"].join("\n\n"), + ); await manager.sync({ force: true }); - let status = manager.status(); + status = manager.status(); expect(status.batch?.enabled).toBe(true); expect(status.batch?.failures).toBe(1); - embedBatch.mockClear(); await fs.writeFile( - path.join(workspaceDir, "memory", "2026-01-10.md"), - ["repeat", "failures", "again"].join("\n\n"), + path.join(workspaceDir, "memory", "2026-01-09.md"), + ["flaky", "batch", "fail-b"].join("\n\n"), ); await manager.sync({ force: true }); status = manager.status(); expect(status.batch?.enabled).toBe(false); expect(status.batch?.failures).toBeGreaterThanOrEqual(2); + // Once disabled, batch endpoints are skipped and fallback embeddings run directly. const fetchCalls = fetchMock.mock.calls.length; embedBatch.mockClear(); await fs.writeFile( - path.join(workspaceDir, "memory", "2026-01-10.md"), - ["repeat", "failures", "fallback"].join("\n\n"), + path.join(workspaceDir, "memory", "2026-01-09.md"), + ["flaky", "batch", "fallback"].join("\n\n"), ); await manager.sync({ force: true }); expect(fetchMock.mock.calls.length).toBe(fetchCalls); diff --git a/src/memory/manager.embedding-batches.test.ts b/src/memory/manager.embedding-batches.test.ts index 3c4019d366b..99cceee162d 100644 --- a/src/memory/manager.embedding-batches.test.ts +++ b/src/memory/manager.embedding-batches.test.ts @@ -1,7 +1,7 @@ import fs from "node:fs/promises"; import os from "node:os"; import path from "node:path"; -import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; import { getMemorySearchManager, type MemoryIndexManager } from "./index.js"; const embedBatch = vi.fn(async (texts: string[]) => texts.map(() => [0, 1, 0])); @@ -20,16 +20,26 @@ vi.mock("./embeddings.js", () => ({ })); describe("memory embedding batches", () => { + let fixtureRoot: string; + let caseId = 0; let workspaceDir: string; let indexPath: string; let manager: MemoryIndexManager | null = null; + beforeAll(async () => { + fixtureRoot = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-mem-")); + }); + + afterAll(async () => { + await fs.rm(fixtureRoot, { recursive: true, force: true }); + }); + beforeEach(async () => { embedBatch.mockClear(); embedQuery.mockClear(); - workspaceDir = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-mem-")); + workspaceDir = path.join(fixtureRoot, `case-${++caseId}`); indexPath = path.join(workspaceDir, "index.sqlite"); - await fs.mkdir(path.join(workspaceDir, "memory")); + await fs.mkdir(path.join(workspaceDir, "memory"), { recursive: true }); }); afterEach(async () => { @@ -37,12 +47,11 @@ describe("memory embedding batches", () => { await manager.close(); manager = null; } - await fs.rm(workspaceDir, { recursive: true, force: true }); }); it("splits large files across multiple embedding batches", async () => { const line = "a".repeat(200); - const content = Array.from({ length: 50 }, () => line).join("\n"); + const content = Array.from({ length: 40 }, () => line).join("\n"); await fs.writeFile(path.join(workspaceDir, "memory", "2026-01-03.md"), content); const cfg = { @@ -68,12 +77,23 @@ describe("memory embedding batches", () => { throw new Error("manager missing"); } manager = result.manager; - await manager.sync({ force: true }); + const updates: Array<{ completed: number; total: number; label?: string }> = []; + await manager.sync({ + force: true, + progress: (update) => { + updates.push(update); + }, + }); const status = manager.status(); const totalTexts = embedBatch.mock.calls.reduce((sum, call) => sum + (call[0]?.length ?? 0), 0); expect(totalTexts).toBe(status.chunks); expect(embedBatch.mock.calls.length).toBeGreaterThan(1); + expect(updates.length).toBeGreaterThan(0); + expect(updates.some((update) => update.label?.includes("/"))).toBe(true); + const last = updates[updates.length - 1]; + expect(last?.total).toBeGreaterThan(0); + expect(last?.completed).toBe(last?.total); }); it("keeps small files in a single embedding batch", async () => { @@ -109,118 +129,21 @@ describe("memory embedding batches", () => { expect(embedBatch.mock.calls.length).toBe(1); }); - it("reports sync progress totals", async () => { - const line = "c".repeat(120); - const content = Array.from({ length: 8 }, () => line).join("\n"); - await fs.writeFile(path.join(workspaceDir, "memory", "2026-01-05.md"), content); - - const cfg = { - agents: { - defaults: { - workspace: workspaceDir, - memorySearch: { - provider: "openai", - model: "mock-embed", - store: { path: indexPath }, - chunking: { tokens: 200, overlap: 0 }, - sync: { watch: false, onSessionStart: false, onSearch: false }, - query: { minScore: 0 }, - }, - }, - list: [{ id: "main", default: true }], - }, - }; - - const result = await getMemorySearchManager({ cfg, agentId: "main" }); - expect(result.manager).not.toBeNull(); - if (!result.manager) { - throw new Error("manager missing"); - } - manager = result.manager; - const updates: Array<{ completed: number; total: number; label?: string }> = []; - await manager.sync({ - force: true, - progress: (update) => { - updates.push(update); - }, - }); - - expect(updates.length).toBeGreaterThan(0); - expect(updates.some((update) => update.label?.includes("/"))).toBe(true); - const last = updates[updates.length - 1]; - expect(last?.total).toBeGreaterThan(0); - expect(last?.completed).toBe(last?.total); - }); - - it("retries embeddings on rate limit errors", async () => { + it("retries embeddings on transient rate limit and 5xx errors", async () => { const line = "d".repeat(120); const content = Array.from({ length: 4 }, () => line).join("\n"); await fs.writeFile(path.join(workspaceDir, "memory", "2026-01-06.md"), content); + const transientErrors = [ + "openai embeddings failed: 429 rate limit", + "openai embeddings failed: 502 Bad Gateway (cloudflare)", + ]; let calls = 0; embedBatch.mockImplementation(async (texts: string[]) => { calls += 1; - if (calls < 3) { - throw new Error("openai embeddings failed: 429 rate limit"); - } - return texts.map(() => [0, 1, 0]); - }); - - const realSetTimeout = setTimeout; - const setTimeoutSpy = vi.spyOn(global, "setTimeout").mockImplementation((( - handler: TimerHandler, - timeout?: number, - ...args: unknown[] - ) => { - const delay = typeof timeout === "number" ? timeout : 0; - if (delay > 0 && delay <= 2000) { - return realSetTimeout(handler, 0, ...args); - } - return realSetTimeout(handler, delay, ...args); - }) as typeof setTimeout); - - const cfg = { - agents: { - defaults: { - workspace: workspaceDir, - memorySearch: { - provider: "openai", - model: "mock-embed", - store: { path: indexPath }, - chunking: { tokens: 200, overlap: 0 }, - sync: { watch: false, onSessionStart: false, onSearch: false }, - query: { minScore: 0 }, - }, - }, - list: [{ id: "main", default: true }], - }, - }; - - const result = await getMemorySearchManager({ cfg, agentId: "main" }); - expect(result.manager).not.toBeNull(); - if (!result.manager) { - throw new Error("manager missing"); - } - manager = result.manager; - try { - await manager.sync({ force: true }); - } finally { - setTimeoutSpy.mockRestore(); - } - - expect(calls).toBe(3); - }, 10000); - - it("retries embeddings on transient 5xx errors", async () => { - const line = "e".repeat(120); - const content = Array.from({ length: 4 }, () => line).join("\n"); - await fs.writeFile(path.join(workspaceDir, "memory", "2026-01-08.md"), content); - - let calls = 0; - embedBatch.mockImplementation(async (texts: string[]) => { - calls += 1; - if (calls < 3) { - throw new Error("openai embeddings failed: 502 Bad Gateway (cloudflare)"); + const transient = transientErrors[calls - 1]; + if (transient) { + throw new Error(transient); } return texts.map(() => [0, 1, 0]); }); diff --git a/src/memory/qmd-manager.test.ts b/src/memory/qmd-manager.test.ts index e8396802862..a4877417c23 100644 --- a/src/memory/qmd-manager.test.ts +++ b/src/memory/qmd-manager.test.ts @@ -2,7 +2,7 @@ import { EventEmitter } from "node:events"; import fs from "node:fs/promises"; import os from "node:os"; import path from "node:path"; -import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; const { logWarnMock, logDebugMock, logInfoMock } = vi.hoisted(() => ({ logWarnMock: vi.fn(), @@ -44,6 +44,18 @@ function createMockChild(params?: { autoClose?: boolean; closeDelayMs?: number } return child; } +function emitAndClose( + child: MockChild, + stream: "stdout" | "stderr", + data: string, + code: number = 0, +) { + queueMicrotask(() => { + child[stream].emit("data", data); + child.closeWith(code); + }); +} + vi.mock("../logging/subsystem.js", () => ({ createSubsystemLogger: () => { const logger = { @@ -66,19 +78,30 @@ import { QmdMemoryManager } from "./qmd-manager.js"; const spawnMock = mockedSpawn as unknown as vi.Mock; describe("QmdMemoryManager", () => { + let fixtureRoot: string; + let fixtureCount = 0; let tmpRoot: string; let workspaceDir: string; let stateDir: string; let cfg: OpenClawConfig; const agentId = "main"; + beforeAll(async () => { + fixtureRoot = await fs.mkdtemp(path.join(os.tmpdir(), "qmd-manager-test-fixtures-")); + }); + + afterAll(async () => { + await fs.rm(fixtureRoot, { recursive: true, force: true }); + }); + beforeEach(async () => { spawnMock.mockReset(); spawnMock.mockImplementation(() => createMockChild()); logWarnMock.mockReset(); logDebugMock.mockReset(); logInfoMock.mockReset(); - tmpRoot = await fs.mkdtemp(path.join(os.tmpdir(), "qmd-manager-test-")); + tmpRoot = path.join(fixtureRoot, `case-${fixtureCount++}`); + await fs.mkdir(tmpRoot, { recursive: true }); workspaceDir = path.join(tmpRoot, "workspace"); await fs.mkdir(workspaceDir, { recursive: true }); stateDir = path.join(tmpRoot, "state"); @@ -102,7 +125,6 @@ describe("QmdMemoryManager", () => { afterEach(async () => { vi.useRealTimers(); delete process.env.OPENCLAW_STATE_DIR; - await fs.rm(tmpRoot, { recursive: true, force: true }); }); it("debounces back-to-back sync calls", async () => { @@ -158,14 +180,11 @@ describe("QmdMemoryManager", () => { const createPromise = QmdMemoryManager.create({ cfg, agentId, resolved }); const race = await Promise.race([ createPromise.then(() => "created" as const), - new Promise<"timeout">((resolve) => setTimeout(() => resolve("timeout"), 80)), + new Promise<"timeout">((resolve) => setTimeout(() => resolve("timeout"), 40)), ]); expect(race).toBe("created"); - - if (!releaseUpdate) { - throw new Error("update child missing"); - } - releaseUpdate(); + await waitForCondition(() => releaseUpdate !== null, 200); + releaseUpdate?.(); const manager = await createPromise; await manager?.close(); }); @@ -202,14 +221,11 @@ describe("QmdMemoryManager", () => { const createPromise = QmdMemoryManager.create({ cfg, agentId, resolved }); const race = await Promise.race([ createPromise.then(() => "created" as const), - new Promise<"timeout">((resolve) => setTimeout(() => resolve("timeout"), 80)), + new Promise<"timeout">((resolve) => setTimeout(() => resolve("timeout"), 40)), ]); expect(race).toBe("timeout"); - - if (!releaseUpdate) { - throw new Error("update child missing"); - } - releaseUpdate(); + await waitForCondition(() => releaseUpdate !== null, 200); + releaseUpdate?.(); const manager = await createPromise; await manager?.close(); }); @@ -301,10 +317,7 @@ describe("QmdMemoryManager", () => { spawnMock.mockImplementation((_cmd: string, args: string[]) => { if (args[0] === "search") { const child = createMockChild({ autoClose: false }); - setTimeout(() => { - child.stdout.emit("data", "[]"); - child.closeWith(0); - }, 0); + emitAndClose(child, "stdout", "[]"); return child; } return createMockChild(); @@ -348,18 +361,12 @@ describe("QmdMemoryManager", () => { spawnMock.mockImplementation((_cmd: string, args: string[]) => { if (args[0] === "search") { const child = createMockChild({ autoClose: false }); - setTimeout(() => { - child.stderr.emit("data", "unknown flag: --json"); - child.closeWith(2); - }, 0); + emitAndClose(child, "stderr", "unknown flag: --json", 2); return child; } if (args[0] === "query") { const child = createMockChild({ autoClose: false }); - setTimeout(() => { - child.stdout.emit("data", "[]"); - child.closeWith(0); - }, 0); + emitAndClose(child, "stdout", "[]"); return child; } return createMockChild(); @@ -435,7 +442,7 @@ describe("QmdMemoryManager", () => { const inFlight = manager.sync({ reason: "interval" }); const forced = manager.sync({ reason: "manual", force: true }); - await new Promise((resolve) => setTimeout(resolve, 20)); + await waitForCondition(() => updateCalls >= 1, 80); expect(updateCalls).toBe(1); if (!releaseFirstUpdate) { throw new Error("first update release missing"); @@ -496,14 +503,14 @@ describe("QmdMemoryManager", () => { const inFlight = manager.sync({ reason: "interval" }); const forcedOne = manager.sync({ reason: "manual", force: true }); - await new Promise((resolve) => setTimeout(resolve, 20)); + await waitForCondition(() => updateCalls >= 1, 80); expect(updateCalls).toBe(1); if (!releaseFirstUpdate) { throw new Error("first update release missing"); } releaseFirstUpdate(); - await waitForCondition(() => updateCalls >= 2, 200); + await waitForCondition(() => updateCalls >= 2, 120); const forcedTwo = manager.sync({ reason: "manual-again", force: true }); if (!releaseSecondUpdate) { @@ -535,10 +542,7 @@ describe("QmdMemoryManager", () => { spawnMock.mockImplementation((_cmd: string, args: string[]) => { if (args[0] === "query") { const child = createMockChild({ autoClose: false }); - setTimeout(() => { - child.stdout.emit("data", "[]"); - child.closeWith(0); - }, 0); + emitAndClose(child, "stdout", "[]"); return child; } return createMockChild(); @@ -805,13 +809,11 @@ describe("QmdMemoryManager", () => { spawnMock.mockImplementation((_cmd: string, args: string[]) => { if (args[0] === "query") { const child = createMockChild({ autoClose: false }); - setTimeout(() => { - child.stdout.emit( - "data", - JSON.stringify([{ docid: "abc123", score: 1, snippet: "@@ -1,1\nremember this" }]), - ); - child.closeWith(0); - }, 0); + emitAndClose( + child, + "stdout", + JSON.stringify([{ docid: "abc123", score: 1, snippet: "@@ -1,1\nremember this" }]), + ); return child; } return createMockChild(); @@ -844,10 +846,7 @@ describe("QmdMemoryManager", () => { spawnMock.mockImplementation((_cmd: string, args: string[]) => { if (args[0] === "query") { const child = createMockChild({ autoClose: false }); - setTimeout(() => { - child.stdout.emit("data", "No results found."); - child.closeWith(0); - }, 0); + emitAndClose(child, "stdout", "No results found."); return child; } return createMockChild(); @@ -870,10 +869,7 @@ describe("QmdMemoryManager", () => { spawnMock.mockImplementation((_cmd: string, args: string[]) => { if (args[0] === "query") { const child = createMockChild({ autoClose: false }); - setTimeout(() => { - child.stdout.emit("data", "No results found\n\n"); - child.closeWith(0); - }, 0); + emitAndClose(child, "stdout", "No results found\n\n"); return child; } return createMockChild(); @@ -896,10 +892,7 @@ describe("QmdMemoryManager", () => { spawnMock.mockImplementation((_cmd: string, args: string[]) => { if (args[0] === "query") { const child = createMockChild({ autoClose: false }); - setTimeout(() => { - child.stderr.emit("data", "No results found.\n"); - child.closeWith(0); - }, 0); + emitAndClose(child, "stderr", "No results found.\n"); return child; } return createMockChild(); @@ -922,11 +915,11 @@ describe("QmdMemoryManager", () => { spawnMock.mockImplementation((_cmd: string, args: string[]) => { if (args[0] === "query") { const child = createMockChild({ autoClose: false }); - setTimeout(() => { + queueMicrotask(() => { child.stdout.emit("data", " \n"); child.stderr.emit("data", "unexpected parser error"); child.closeWith(0); - }, 0); + }); return child; } return createMockChild(); @@ -1034,7 +1027,7 @@ async function waitForCondition(check: () => boolean, timeoutMs: number): Promis if (check()) { return; } - await new Promise((resolve) => setTimeout(resolve, 5)); + await new Promise((resolve) => setTimeout(resolve, 2)); } throw new Error("condition was not met in time"); } diff --git a/src/plugins/hooks.ts b/src/plugins/hooks.ts index d74c23c5b21..040ce1d35c8 100644 --- a/src/plugins/hooks.ts +++ b/src/plugins/hooks.ts @@ -14,6 +14,7 @@ import type { PluginHookBeforeAgentStartEvent, PluginHookBeforeAgentStartResult, PluginHookBeforeCompactionEvent, + PluginHookBeforeResetEvent, PluginHookBeforeToolCallEvent, PluginHookBeforeToolCallResult, PluginHookGatewayContext, @@ -42,6 +43,7 @@ export type { PluginHookBeforeAgentStartResult, PluginHookAgentEndEvent, PluginHookBeforeCompactionEvent, + PluginHookBeforeResetEvent, PluginHookAfterCompactionEvent, PluginHookMessageContext, PluginHookMessageReceivedEvent, @@ -230,6 +232,18 @@ export function createHookRunner(registry: PluginRegistry, options: HookRunnerOp return runVoidHook("after_compaction", event, ctx); } + /** + * Run before_reset hook. + * Fired when /new or /reset clears a session, before messages are lost. + * Runs in parallel (fire-and-forget). + */ + async function runBeforeReset( + event: PluginHookBeforeResetEvent, + ctx: PluginHookAgentContext, + ): Promise { + return runVoidHook("before_reset", event, ctx); + } + // ========================================================================= // Message Hooks // ========================================================================= @@ -447,6 +461,7 @@ export function createHookRunner(registry: PluginRegistry, options: HookRunnerOp runAgentEnd, runBeforeCompaction, runAfterCompaction, + runBeforeReset, // Message hooks runMessageReceived, runMessageSending, diff --git a/src/plugins/install.e2e.test.ts b/src/plugins/install.e2e.test.ts index b81d7fc5638..c603dd9d97f 100644 --- a/src/plugins/install.e2e.test.ts +++ b/src/plugins/install.e2e.test.ts @@ -1,9 +1,9 @@ import JSZip from "jszip"; -import { spawnSync } from "node:child_process"; import { randomUUID } from "node:crypto"; import fs from "node:fs"; import os from "node:os"; import path from "node:path"; +import * as tar from "tar"; import { afterEach, describe, expect, it, vi } from "vitest"; import * as skillScanner from "../security/skill-scanner.js"; @@ -20,40 +20,7 @@ function makeTempDir() { return dir; } -function resolveNpmCliJs() { - const fromEnv = process.env.npm_execpath; - if (fromEnv?.includes(`${path.sep}npm${path.sep}`) && fromEnv?.endsWith("npm-cli.js")) { - return fromEnv ?? null; - } - - const fromNodeDir = path.join( - path.dirname(process.execPath), - "node_modules", - "npm", - "bin", - "npm-cli.js", - ); - if (fs.existsSync(fromNodeDir)) { - return fromNodeDir; - } - - const fromLibNodeModules = path.resolve( - path.dirname(process.execPath), - "..", - "lib", - "node_modules", - "npm", - "bin", - "npm-cli.js", - ); - if (fs.existsSync(fromLibNodeModules)) { - return fromLibNodeModules; - } - - return null; -} - -function packToArchive({ +async function packToArchive({ pkgDir, outDir, outName, @@ -62,27 +29,16 @@ function packToArchive({ outDir: string; outName: string; }) { - const npmCli = resolveNpmCliJs(); - const cmd = npmCli ? process.execPath : "npm"; - const args = npmCli - ? [npmCli, "pack", "--silent", "--pack-destination", outDir, pkgDir] - : ["pack", "--silent", "--pack-destination", outDir, pkgDir]; - - const res = spawnSync(cmd, args, { encoding: "utf-8" }); - expect(res.status).toBe(0); - if (res.status !== 0) { - throw new Error(`npm pack failed: ${res.stderr || res.stdout || ""}`); - } - - const packed = (res.stdout || "").trim().split(/\r?\n/).filter(Boolean).at(-1); - if (!packed) { - throw new Error(`npm pack did not output a filename: ${res.stdout || ""}`); - } - - const src = path.join(outDir, packed); const dest = path.join(outDir, outName); fs.rmSync(dest, { force: true }); - fs.renameSync(src, dest); + await tar.c( + { + gzip: true, + file: dest, + cwd: path.dirname(pkgDir), + }, + [path.basename(pkgDir)], + ); return dest; } @@ -113,7 +69,7 @@ describe("installPluginFromArchive", () => { ); fs.writeFileSync(path.join(pkgDir, "dist", "index.js"), "export {};", "utf-8"); - const archivePath = packToArchive({ + const archivePath = await packToArchive({ pkgDir, outDir: workDir, outName: "plugin.tgz", @@ -151,7 +107,7 @@ describe("installPluginFromArchive", () => { ); fs.writeFileSync(path.join(pkgDir, "dist", "index.js"), "export {};", "utf-8"); - const archivePath = packToArchive({ + const archivePath = await packToArchive({ pkgDir, outDir: workDir, outName: "plugin.tgz", @@ -227,13 +183,13 @@ describe("installPluginFromArchive", () => { ); fs.writeFileSync(path.join(pkgDir, "dist", "index.js"), "export {};", "utf-8"); - const archiveV1 = packToArchive({ + const archiveV1 = await packToArchive({ pkgDir, outDir: workDir, outName: "plugin-v1.tgz", }); - const archiveV2 = (() => { + const archiveV2 = await (async () => { fs.writeFileSync( path.join(pkgDir, "package.json"), JSON.stringify({ @@ -243,7 +199,7 @@ describe("installPluginFromArchive", () => { }), "utf-8", ); - return packToArchive({ + return await packToArchive({ pkgDir, outDir: workDir, outName: "plugin-v2.tgz", @@ -289,7 +245,7 @@ describe("installPluginFromArchive", () => { ); fs.writeFileSync(path.join(pkgDir, "dist", "index.js"), "export {};", "utf-8"); - const archivePath = packToArchive({ + const archivePath = await packToArchive({ pkgDir, outDir: workDir, outName: "traversal.tgz", @@ -325,7 +281,7 @@ describe("installPluginFromArchive", () => { ); fs.writeFileSync(path.join(pkgDir, "dist", "index.js"), "export {};", "utf-8"); - const archivePath = packToArchive({ + const archivePath = await packToArchive({ pkgDir, outDir: workDir, outName: "reserved.tgz", @@ -356,7 +312,7 @@ describe("installPluginFromArchive", () => { "utf-8", ); - const archivePath = packToArchive({ + const archivePath = await packToArchive({ pkgDir, outDir: workDir, outName: "bad.tgz", diff --git a/src/plugins/loader.test.ts b/src/plugins/loader.test.ts index cd27cc69ef2..f32d04d0d80 100644 --- a/src/plugins/loader.test.ts +++ b/src/plugins/loader.test.ts @@ -2,19 +2,19 @@ import { randomUUID } from "node:crypto"; import fs from "node:fs"; import os from "node:os"; import path from "node:path"; -import { afterEach, describe, expect, it } from "vitest"; +import { afterAll, afterEach, describe, expect, it } from "vitest"; import { loadOpenClawPlugins } from "./loader.js"; type TempPlugin = { dir: string; file: string; id: string }; -const tempDirs: string[] = []; +const fixtureRoot = path.join(os.tmpdir(), `openclaw-plugin-${randomUUID()}`); +let tempDirIndex = 0; const prevBundledDir = process.env.OPENCLAW_BUNDLED_PLUGINS_DIR; const EMPTY_PLUGIN_SCHEMA = { type: "object", additionalProperties: false, properties: {} }; function makeTempDir() { - const dir = path.join(os.tmpdir(), `openclaw-plugin-${randomUUID()}`); + const dir = path.join(fixtureRoot, `case-${tempDirIndex++}`); fs.mkdirSync(dir, { recursive: true }); - tempDirs.push(dir); return dir; } @@ -44,13 +44,6 @@ function writePlugin(params: { } afterEach(() => { - for (const dir of tempDirs.splice(0)) { - try { - fs.rmSync(dir, { recursive: true, force: true }); - } catch { - // ignore cleanup failures - } - } if (prevBundledDir === undefined) { delete process.env.OPENCLAW_BUNDLED_PLUGINS_DIR; } else { @@ -58,6 +51,14 @@ afterEach(() => { } }); +afterAll(() => { + try { + fs.rmSync(fixtureRoot, { recursive: true, force: true }); + } catch { + // ignore cleanup failures + } +}); + describe("loadOpenClawPlugins", () => { it("disables bundled plugins by default", () => { const bundledDir = makeTempDir(); diff --git a/src/plugins/tools.optional.test.ts b/src/plugins/tools.optional.test.ts index 1f15eec90ea..614c0980179 100644 --- a/src/plugins/tools.optional.test.ts +++ b/src/plugins/tools.optional.test.ts @@ -2,23 +2,22 @@ import { randomUUID } from "node:crypto"; import fs from "node:fs"; import os from "node:os"; import path from "node:path"; -import { afterEach, describe, expect, it } from "vitest"; +import { afterAll, describe, expect, it } from "vitest"; import { resolvePluginTools } from "./tools.js"; type TempPlugin = { dir: string; file: string; id: string }; -const tempDirs: string[] = []; +const fixtureRoot = path.join(os.tmpdir(), `openclaw-plugin-tools-${randomUUID()}`); const EMPTY_PLUGIN_SCHEMA = { type: "object", additionalProperties: false, properties: {} }; -function makeTempDir() { - const dir = path.join(os.tmpdir(), `openclaw-plugin-tools-${randomUUID()}`); +function makeFixtureDir(id: string) { + const dir = path.join(fixtureRoot, id); fs.mkdirSync(dir, { recursive: true }); - tempDirs.push(dir); return dir; } function writePlugin(params: { id: string; body: string }): TempPlugin { - const dir = makeTempDir(); + const dir = makeFixtureDir(params.id); const file = path.join(dir, `${params.id}.js`); fs.writeFileSync(file, params.body, "utf-8"); fs.writeFileSync( @@ -36,18 +35,7 @@ function writePlugin(params: { id: string; body: string }): TempPlugin { return { dir, file, id: params.id }; } -afterEach(() => { - for (const dir of tempDirs.splice(0)) { - try { - fs.rmSync(dir, { recursive: true, force: true }); - } catch { - // ignore cleanup failures - } - } -}); - -describe("resolvePluginTools optional tools", () => { - const pluginBody = ` +const pluginBody = ` export default { register(api) { api.registerTool( { @@ -63,92 +51,11 @@ export default { register(api) { } } `; - it("skips optional tools without explicit allowlist", () => { - const plugin = writePlugin({ id: "optional-demo", body: pluginBody }); - const tools = resolvePluginTools({ - context: { - config: { - plugins: { - load: { paths: [plugin.file] }, - allow: [plugin.id], - }, - }, - workspaceDir: plugin.dir, - }, - }); - expect(tools).toHaveLength(0); - }); - - it("allows optional tools by name", () => { - const plugin = writePlugin({ id: "optional-demo", body: pluginBody }); - const tools = resolvePluginTools({ - context: { - config: { - plugins: { - load: { paths: [plugin.file] }, - allow: [plugin.id], - }, - }, - workspaceDir: plugin.dir, - }, - toolAllowlist: ["optional_tool"], - }); - expect(tools.map((tool) => tool.name)).toContain("optional_tool"); - }); - - it("allows optional tools via plugin groups", () => { - const plugin = writePlugin({ id: "optional-demo", body: pluginBody }); - const toolsAll = resolvePluginTools({ - context: { - config: { - plugins: { - load: { paths: [plugin.file] }, - allow: [plugin.id], - }, - }, - workspaceDir: plugin.dir, - }, - toolAllowlist: ["group:plugins"], - }); - expect(toolsAll.map((tool) => tool.name)).toContain("optional_tool"); - - const toolsPlugin = resolvePluginTools({ - context: { - config: { - plugins: { - load: { paths: [plugin.file] }, - allow: [plugin.id], - }, - }, - workspaceDir: plugin.dir, - }, - toolAllowlist: ["optional-demo"], - }); - expect(toolsPlugin.map((tool) => tool.name)).toContain("optional_tool"); - }); - - it("rejects plugin id collisions with core tool names", () => { - const plugin = writePlugin({ id: "message", body: pluginBody }); - const tools = resolvePluginTools({ - context: { - config: { - plugins: { - load: { paths: [plugin.file] }, - allow: [plugin.id], - }, - }, - workspaceDir: plugin.dir, - }, - existingToolNames: new Set(["message"]), - toolAllowlist: ["message"], - }); - expect(tools).toHaveLength(0); - }); - - it("skips conflicting tool names but keeps other tools", () => { - const plugin = writePlugin({ - id: "multi", - body: ` +const optionalDemoPlugin = writePlugin({ id: "optional-demo", body: pluginBody }); +const coreNameCollisionPlugin = writePlugin({ id: "message", body: pluginBody }); +const multiToolPlugin = writePlugin({ + id: "multi", + body: ` export default { register(api) { api.registerTool({ name: "message", @@ -168,17 +75,105 @@ export default { register(api) { }); } } `, - }); +}); +afterAll(() => { + try { + fs.rmSync(fixtureRoot, { recursive: true, force: true }); + } catch { + // ignore cleanup failures + } +}); + +describe("resolvePluginTools optional tools", () => { + it("skips optional tools without explicit allowlist", () => { const tools = resolvePluginTools({ context: { config: { plugins: { - load: { paths: [plugin.file] }, - allow: [plugin.id], + load: { paths: [optionalDemoPlugin.file] }, + allow: [optionalDemoPlugin.id], }, }, - workspaceDir: plugin.dir, + workspaceDir: optionalDemoPlugin.dir, + }, + }); + expect(tools).toHaveLength(0); + }); + + it("allows optional tools by name", () => { + const tools = resolvePluginTools({ + context: { + config: { + plugins: { + load: { paths: [optionalDemoPlugin.file] }, + allow: [optionalDemoPlugin.id], + }, + }, + workspaceDir: optionalDemoPlugin.dir, + }, + toolAllowlist: ["optional_tool"], + }); + expect(tools.map((tool) => tool.name)).toContain("optional_tool"); + }); + + it("allows optional tools via plugin groups", () => { + const toolsAll = resolvePluginTools({ + context: { + config: { + plugins: { + load: { paths: [optionalDemoPlugin.file] }, + allow: [optionalDemoPlugin.id], + }, + }, + workspaceDir: optionalDemoPlugin.dir, + }, + toolAllowlist: ["group:plugins"], + }); + expect(toolsAll.map((tool) => tool.name)).toContain("optional_tool"); + + const toolsPlugin = resolvePluginTools({ + context: { + config: { + plugins: { + load: { paths: [optionalDemoPlugin.file] }, + allow: [optionalDemoPlugin.id], + }, + }, + workspaceDir: optionalDemoPlugin.dir, + }, + toolAllowlist: ["optional-demo"], + }); + expect(toolsPlugin.map((tool) => tool.name)).toContain("optional_tool"); + }); + + it("rejects plugin id collisions with core tool names", () => { + const tools = resolvePluginTools({ + context: { + config: { + plugins: { + load: { paths: [coreNameCollisionPlugin.file] }, + allow: [coreNameCollisionPlugin.id], + }, + }, + workspaceDir: coreNameCollisionPlugin.dir, + }, + existingToolNames: new Set(["message"]), + toolAllowlist: ["message"], + }); + expect(tools).toHaveLength(0); + }); + + it("skips conflicting tool names but keeps other tools", () => { + const tools = resolvePluginTools({ + context: { + config: { + plugins: { + load: { paths: [multiToolPlugin.file] }, + allow: [multiToolPlugin.id], + }, + }, + workspaceDir: multiToolPlugin.dir, }, existingToolNames: new Set(["message"]), }); diff --git a/src/plugins/types.ts b/src/plugins/types.ts index 27c6fff2425..32a961df6e6 100644 --- a/src/plugins/types.ts +++ b/src/plugins/types.ts @@ -300,6 +300,7 @@ export type PluginHookName = | "agent_end" | "before_compaction" | "after_compaction" + | "before_reset" | "message_received" | "message_sending" | "message_sent" @@ -315,6 +316,7 @@ export type PluginHookName = export type PluginHookAgentContext = { agentId?: string; sessionKey?: string; + sessionId?: string; workspaceDir?: string; messageProvider?: string; }; @@ -340,14 +342,33 @@ export type PluginHookAgentEndEvent = { // Compaction hooks export type PluginHookBeforeCompactionEvent = { + /** Total messages in the session before any truncation or compaction */ messageCount: number; + /** Messages being fed to the compaction LLM (after history-limit truncation) */ + compactingCount?: number; tokenCount?: number; + messages?: unknown[]; + /** Path to the session JSONL transcript. All messages are already on disk + * before compaction starts, so plugins can read this file asynchronously + * and process in parallel with the compaction LLM call. */ + sessionFile?: string; +}; + +// before_reset hook β€” fired when /new or /reset clears a session +export type PluginHookBeforeResetEvent = { + sessionFile?: string; + messages?: unknown[]; + reason?: string; }; export type PluginHookAfterCompactionEvent = { messageCount: number; tokenCount?: number; compactedCount: number; + /** Path to the session JSONL transcript. All pre-compaction messages are + * preserved on disk, so plugins can read and process them asynchronously + * without blocking the compaction pipeline. */ + sessionFile?: string; }; // Message context @@ -486,6 +507,10 @@ export type PluginHookHandlerMap = { event: PluginHookAfterCompactionEvent, ctx: PluginHookAgentContext, ) => Promise | void; + before_reset: ( + event: PluginHookBeforeResetEvent, + ctx: PluginHookAgentContext, + ) => Promise | void; message_received: ( event: PluginHookMessageReceivedEvent, ctx: PluginHookMessageContext, diff --git a/src/plugins/wired-hooks-after-tool-call.e2e.test.ts b/src/plugins/wired-hooks-after-tool-call.e2e.test.ts index 0256f6f3b62..cddbf3b42ba 100644 --- a/src/plugins/wired-hooks-after-tool-call.e2e.test.ts +++ b/src/plugins/wired-hooks-after-tool-call.e2e.test.ts @@ -6,6 +6,7 @@ import { beforeEach, describe, expect, it, vi } from "vitest"; const hookMocks = vi.hoisted(() => ({ runner: { hasHooks: vi.fn(() => false), + runBeforeToolCall: vi.fn(async () => {}), runAfterToolCall: vi.fn(async () => {}), }, })); @@ -23,6 +24,8 @@ describe("after_tool_call hook wiring", () => { beforeEach(() => { hookMocks.runner.hasHooks.mockReset(); hookMocks.runner.hasHooks.mockReturnValue(false); + hookMocks.runner.runBeforeToolCall.mockReset(); + hookMocks.runner.runBeforeToolCall.mockResolvedValue(undefined); hookMocks.runner.runAfterToolCall.mockReset(); hookMocks.runner.runAfterToolCall.mockResolvedValue(undefined); }); @@ -84,6 +87,7 @@ describe("after_tool_call hook wiring", () => { ); expect(hookMocks.runner.runAfterToolCall).toHaveBeenCalledTimes(1); + expect(hookMocks.runner.runBeforeToolCall).not.toHaveBeenCalled(); const [event, context] = hookMocks.runner.runAfterToolCall.mock.calls[0]; expect(event.toolName).toBe("read"); diff --git a/src/process/child-process-bridge.test.ts b/src/process/child-process-bridge.test.ts index 0a37ac7504a..855b37ac2ea 100644 --- a/src/process/child-process-bridge.test.ts +++ b/src/process/child-process-bridge.test.ts @@ -51,6 +51,17 @@ function canConnect(port: number): Promise { }); } +async function waitForPortClosed(port: number, timeoutMs = 1_000): Promise { + const deadline = Date.now() + timeoutMs; + while (Date.now() <= deadline) { + if (!(await canConnect(port))) { + return; + } + await new Promise((resolve) => setTimeout(resolve, 10)); + } + throw new Error("timeout waiting for port to close"); +} + describe("attachChildProcessBridge", () => { const children: Array<{ kill: (signal?: NodeJS.Signals) => boolean }> = []; const detachments: Array<() => void> = []; @@ -111,7 +122,7 @@ describe("attachChildProcessBridge", () => { }); }); - await new Promise((r) => setTimeout(r, 250)); + await waitForPortClosed(port); expect(await canConnect(port)).toBe(false); }, 20_000); }); diff --git a/src/process/command-queue.test.ts b/src/process/command-queue.test.ts index 60034b43929..79b8389a8b5 100644 --- a/src/process/command-queue.test.ts +++ b/src/process/command-queue.test.ts @@ -23,6 +23,7 @@ import { enqueueCommandInLane, getActiveTaskCount, getQueueSize, + resetAllLanes, setCommandLaneConcurrency, waitForActiveTasks, } from "./command-queue.js"; @@ -36,6 +37,12 @@ describe("command queue", () => { diagnosticMocks.diag.error.mockClear(); }); + it("resetAllLanes is safe when no lanes have been created", () => { + expect(getActiveTaskCount()).toBe(0); + expect(() => resetAllLanes()).not.toThrow(); + expect(getActiveTaskCount()).toBe(0); + }); + it("runs tasks one at a time in order", async () => { let active = 0; let maxActive = 0; @@ -105,8 +112,6 @@ describe("command queue", () => { await blocker; }); - // Give the event loop a tick for the task to start. - await new Promise((r) => setTimeout(r, 5)); expect(getActiveTaskCount()).toBe(1); resolve1(); @@ -129,18 +134,21 @@ describe("command queue", () => { await blocker; }); - // Give the task a tick to start. - await new Promise((r) => setTimeout(r, 5)); + vi.useFakeTimers(); + try { + const drainPromise = waitForActiveTasks(5000); - const drainPromise = waitForActiveTasks(5000); + // Resolve the blocker after a short delay. + setTimeout(() => resolve1(), 10); + await vi.advanceTimersByTimeAsync(100); - // Resolve the blocker after a short delay. - setTimeout(() => resolve1(), 50); + const { drained } = await drainPromise; + expect(drained).toBe(true); - const { drained } = await drainPromise; - expect(drained).toBe(true); - - await task; + await task; + } finally { + vi.useRealTimers(); + } }); it("waitForActiveTasks returns drained=false on timeout", async () => { @@ -153,13 +161,61 @@ describe("command queue", () => { await blocker; }); - await new Promise((r) => setTimeout(r, 5)); + vi.useFakeTimers(); + try { + const waitPromise = waitForActiveTasks(50); + await vi.advanceTimersByTimeAsync(100); + const { drained } = await waitPromise; + expect(drained).toBe(false); - const { drained } = await waitForActiveTasks(50); - expect(drained).toBe(false); + resolve1(); + await task; + } finally { + vi.useRealTimers(); + } + }); + it("resetAllLanes drains queued work immediately after reset", async () => { + const lane = `reset-test-${Date.now()}-${Math.random().toString(16).slice(2)}`; + setCommandLaneConcurrency(lane, 1); + + let resolve1!: () => void; + const blocker = new Promise((r) => { + resolve1 = r; + }); + + // Start a task that blocks the lane + const task1 = enqueueCommandInLane(lane, async () => { + await blocker; + }); + + await vi.waitFor(() => { + expect(getActiveTaskCount()).toBeGreaterThanOrEqual(1); + }); + + // Enqueue another task β€” it should be stuck behind the blocker + let task2Ran = false; + const task2 = enqueueCommandInLane(lane, async () => { + task2Ran = true; + }); + + await vi.waitFor(() => { + expect(getQueueSize(lane)).toBeGreaterThanOrEqual(2); + }); + expect(task2Ran).toBe(false); + + // Simulate SIGUSR1: reset all lanes. Queued work (task2) should be + // drained immediately β€” no fresh enqueue needed. + resetAllLanes(); + + // Complete the stale in-flight task; generation mismatch makes its + // completion path a no-op for queue bookkeeping. resolve1(); - await task; + await task1; + + // task2 should have been pumped by resetAllLanes's drain pass. + await task2; + expect(task2Ran).toBe(true); }); it("waitForActiveTasks ignores tasks that start after the call", async () => { @@ -178,15 +234,12 @@ describe("command queue", () => { const first = enqueueCommandInLane(lane, async () => { await blocker1; }); - await new Promise((r) => setTimeout(r, 5)); - const drainPromise = waitForActiveTasks(2000); // Starts after waitForActiveTasks snapshot and should not block drain completion. const second = enqueueCommandInLane(lane, async () => { await blocker2; }); - await new Promise((r) => setTimeout(r, 5)); expect(getActiveTaskCount()).toBeGreaterThanOrEqual(2); resolve1(); @@ -212,9 +265,6 @@ describe("command queue", () => { // Second task is queued behind the first. const second = enqueueCommand(async () => "second"); - // Give the first task a tick to start. - await new Promise((r) => setTimeout(r, 5)); - const removed = clearCommandLane(); expect(removed).toBe(1); // only the queued (not active) entry diff --git a/src/process/command-queue.ts b/src/process/command-queue.ts index b0f012ca245..9ee4c741719 100644 --- a/src/process/command-queue.ts +++ b/src/process/command-queue.ts @@ -29,10 +29,10 @@ type QueueEntry = { type LaneState = { lane: string; queue: QueueEntry[]; - active: number; activeTaskIds: Set; maxConcurrent: number; draining: boolean; + generation: number; }; const lanes = new Map(); @@ -46,15 +46,23 @@ function getLaneState(lane: string): LaneState { const created: LaneState = { lane, queue: [], - active: 0, activeTaskIds: new Set(), maxConcurrent: 1, draining: false, + generation: 0, }; lanes.set(lane, created); return created; } +function completeTask(state: LaneState, taskId: number, taskGeneration: number): boolean { + if (taskGeneration !== state.generation) { + return false; + } + state.activeTaskIds.delete(taskId); + return true; +} + function drainLane(lane: string) { const state = getLaneState(lane); if (state.draining) { @@ -63,7 +71,7 @@ function drainLane(lane: string) { state.draining = true; const pump = () => { - while (state.active < state.maxConcurrent && state.queue.length > 0) { + while (state.activeTaskIds.size < state.maxConcurrent && state.queue.length > 0) { const entry = state.queue.shift() as QueueEntry; const waitedMs = Date.now() - entry.enqueuedAt; if (waitedMs >= entry.warnAfterMs) { @@ -74,29 +82,31 @@ function drainLane(lane: string) { } logLaneDequeue(lane, waitedMs, state.queue.length); const taskId = nextTaskId++; - state.active += 1; + const taskGeneration = state.generation; state.activeTaskIds.add(taskId); void (async () => { const startTime = Date.now(); try { const result = await entry.task(); - state.active -= 1; - state.activeTaskIds.delete(taskId); - diag.debug( - `lane task done: lane=${lane} durationMs=${Date.now() - startTime} active=${state.active} queued=${state.queue.length}`, - ); - pump(); + const completedCurrentGeneration = completeTask(state, taskId, taskGeneration); + if (completedCurrentGeneration) { + diag.debug( + `lane task done: lane=${lane} durationMs=${Date.now() - startTime} active=${state.activeTaskIds.size} queued=${state.queue.length}`, + ); + pump(); + } entry.resolve(result); } catch (err) { - state.active -= 1; - state.activeTaskIds.delete(taskId); + const completedCurrentGeneration = completeTask(state, taskId, taskGeneration); const isProbeLane = lane.startsWith("auth-probe:") || lane.startsWith("session:probe-"); if (!isProbeLane) { diag.error( `lane task error: lane=${lane} durationMs=${Date.now() - startTime} error="${String(err)}"`, ); } - pump(); + if (completedCurrentGeneration) { + pump(); + } entry.reject(err); } })(); @@ -134,7 +144,7 @@ export function enqueueCommandInLane( warnAfterMs, onWait: opts?.onWait, }); - logLaneEnqueue(cleaned, state.queue.length + state.active); + logLaneEnqueue(cleaned, state.queue.length + state.activeTaskIds.size); drainLane(cleaned); }); } @@ -155,13 +165,13 @@ export function getQueueSize(lane: string = CommandLane.Main) { if (!state) { return 0; } - return state.queue.length + state.active; + return state.queue.length + state.activeTaskIds.size; } export function getTotalQueueSize() { let total = 0; for (const s of lanes.values()) { - total += s.queue.length + s.active; + total += s.queue.length + s.activeTaskIds.size; } return total; } @@ -180,6 +190,36 @@ export function clearCommandLane(lane: string = CommandLane.Main) { return removed; } +/** + * Reset all lane runtime state to idle. Used after SIGUSR1 in-process + * restarts where interrupted tasks' finally blocks may not run, leaving + * stale active task IDs that permanently block new work from draining. + * + * Bumps lane generation and clears execution counters so stale completions + * from old in-flight tasks are ignored. Queued entries are intentionally + * preserved β€” they represent pending user work that should still execute + * after restart. + * + * After resetting, drains any lanes that still have queued entries so + * preserved work is pumped immediately rather than waiting for a future + * `enqueueCommandInLane()` call (which may never come). + */ +export function resetAllLanes(): void { + const lanesToDrain: string[] = []; + for (const state of lanes.values()) { + state.generation += 1; + state.activeTaskIds.clear(); + state.draining = false; + if (state.queue.length > 0) { + lanesToDrain.push(state.lane); + } + } + // Drain after the full reset pass so all lanes are in a clean state first. + for (const lane of lanesToDrain) { + drainLane(lane); + } +} + /** * Returns the total number of actively executing tasks across all lanes * (excludes queued-but-not-started entries). @@ -187,7 +227,7 @@ export function clearCommandLane(lane: string = CommandLane.Main) { export function getActiveTaskCount(): number { let total = 0; for (const s of lanes.values()) { - total += s.active; + total += s.activeTaskIds.size; } return total; } diff --git a/src/process/restart-recovery.test.ts b/src/process/restart-recovery.test.ts new file mode 100644 index 00000000000..5091d7b9928 --- /dev/null +++ b/src/process/restart-recovery.test.ts @@ -0,0 +1,18 @@ +import { describe, expect, it, vi } from "vitest"; +import { createRestartIterationHook } from "./restart-recovery.js"; + +describe("restart-recovery", () => { + it("skips recovery on first iteration and runs on subsequent iterations", () => { + const onRestart = vi.fn(); + const onIteration = createRestartIterationHook(onRestart); + + expect(onIteration()).toBe(false); + expect(onRestart).not.toHaveBeenCalled(); + + expect(onIteration()).toBe(true); + expect(onRestart).toHaveBeenCalledTimes(1); + + expect(onIteration()).toBe(true); + expect(onRestart).toHaveBeenCalledTimes(2); + }); +}); diff --git a/src/process/restart-recovery.ts b/src/process/restart-recovery.ts new file mode 100644 index 00000000000..2f9818d7f5a --- /dev/null +++ b/src/process/restart-recovery.ts @@ -0,0 +1,16 @@ +/** + * Returns an iteration hook for in-process restart loops. + * The first call is considered initial startup and does nothing. + * Each subsequent call represents a restart iteration and invokes `onRestart`. + */ +export function createRestartIterationHook(onRestart: () => void): () => boolean { + let isFirstIteration = true; + return () => { + if (isFirstIteration) { + isFirstIteration = false; + return false; + } + onRestart(); + return true; + }; +} diff --git a/src/runtime.ts b/src/runtime.ts index c8eab74ec6a..1c6fcec30ff 100644 --- a/src/runtime.ts +++ b/src/runtime.ts @@ -7,8 +7,22 @@ export type RuntimeEnv = { exit: (code: number) => never; }; +function shouldEmitRuntimeLog(env: NodeJS.ProcessEnv = process.env): boolean { + if (env.VITEST !== "true") { + return true; + } + if (env.OPENCLAW_TEST_RUNTIME_LOG === "1") { + return true; + } + const maybeMockedLog = console.log as unknown as { mock?: unknown }; + return typeof maybeMockedLog.mock === "object"; +} + export const defaultRuntime: RuntimeEnv = { log: (...args: Parameters) => { + if (!shouldEmitRuntimeLog()) { + return; + } clearActiveProgressLine(); console.log(...args); }, @@ -22,3 +36,22 @@ export const defaultRuntime: RuntimeEnv = { throw new Error("unreachable"); // satisfies tests when mocked }, }; + +export function createNonExitingRuntime(): RuntimeEnv { + return { + log: (...args: Parameters) => { + if (!shouldEmitRuntimeLog()) { + return; + } + clearActiveProgressLine(); + console.log(...args); + }, + error: (...args: Parameters) => { + clearActiveProgressLine(); + console.error(...args); + }, + exit: (code: number): never => { + throw new Error(`exit ${code}`); + }, + }; +} diff --git a/src/signal/monitor.ts b/src/signal/monitor.ts index aabe0021b43..85f5cac66d1 100644 --- a/src/signal/monitor.ts +++ b/src/signal/monitor.ts @@ -1,12 +1,12 @@ import type { ReplyPayload } from "../auto-reply/types.js"; import type { OpenClawConfig } from "../config/config.js"; import type { SignalReactionNotificationMode } from "../config/types.js"; -import type { RuntimeEnv } from "../runtime.js"; import { chunkTextWithMode, resolveChunkMode, resolveTextChunkLimit } from "../auto-reply/chunk.js"; import { DEFAULT_GROUP_HISTORY_LIMIT, type HistoryEntry } from "../auto-reply/reply/history.js"; import { loadConfig } from "../config/config.js"; import { waitForTransportReady } from "../infra/transport-ready.js"; import { saveMediaBuffer } from "../media/store.js"; +import { createNonExitingRuntime, type RuntimeEnv } from "../runtime.js"; import { normalizeE164 } from "../utils.js"; import { resolveSignalAccount } from "./accounts.js"; import { signalCheck, signalRpcRequest } from "./client.js"; @@ -57,15 +57,7 @@ export type MonitorSignalOpts = { }; function resolveRuntime(opts: MonitorSignalOpts): RuntimeEnv { - return ( - opts.runtime ?? { - log: console.log, - error: console.error, - exit: (code: number): never => { - throw new Error(`exit ${code}`); - }, - } - ); + return opts.runtime ?? createNonExitingRuntime(); } function normalizeAllowList(raw?: Array): string[] { diff --git a/src/slack/monitor/provider.ts b/src/slack/monitor/provider.ts index 6c544655cca..64768ee0ae3 100644 --- a/src/slack/monitor/provider.ts +++ b/src/slack/monitor/provider.ts @@ -1,7 +1,6 @@ import type { IncomingMessage, ServerResponse } from "node:http"; import SlackBolt from "@slack/bolt"; import type { SessionScope } from "../../config/sessions.js"; -import type { RuntimeEnv } from "../../runtime.js"; import type { MonitorSlackOpts } from "./types.js"; import { resolveTextChunkLimit } from "../../auto-reply/chunk.js"; import { DEFAULT_GROUP_HISTORY_LIMIT } from "../../auto-reply/reply/history.js"; @@ -10,6 +9,7 @@ import { loadConfig } from "../../config/config.js"; import { warn } from "../../globals.js"; import { installRequestBodyLimitGuard } from "../../infra/http-body.js"; import { normalizeMainKey } from "../../routing/session-key.js"; +import { createNonExitingRuntime, type RuntimeEnv } from "../../runtime.js"; import { resolveSlackAccount } from "../accounts.js"; import { resolveSlackWebClientOptions } from "../client.js"; import { normalizeSlackWebhookPath, registerSlackHttpHandler } from "../http/index.js"; @@ -81,13 +81,7 @@ export async function monitorSlackProvider(opts: MonitorSlackOpts = {}) { ); } - const runtime: RuntimeEnv = opts.runtime ?? { - log: console.log, - error: console.error, - exit: (code: number): never => { - throw new Error(`exit ${code}`); - }, - }; + const runtime: RuntimeEnv = opts.runtime ?? createNonExitingRuntime(); const slackCfg = account.config; const dmConfig = slackCfg.dm; diff --git a/src/telegram/bot-native-command-menu.test.ts b/src/telegram/bot-native-command-menu.test.ts new file mode 100644 index 00000000000..a1b77e94384 --- /dev/null +++ b/src/telegram/bot-native-command-menu.test.ts @@ -0,0 +1,79 @@ +import { describe, expect, it, vi } from "vitest"; +import { + buildCappedTelegramMenuCommands, + buildPluginTelegramMenuCommands, + syncTelegramMenuCommands, +} from "./bot-native-command-menu.js"; + +describe("bot-native-command-menu", () => { + it("caps menu entries to Telegram limit", () => { + const allCommands = Array.from({ length: 105 }, (_, i) => ({ + command: `cmd_${i}`, + description: `Command ${i}`, + })); + + const result = buildCappedTelegramMenuCommands({ allCommands }); + + expect(result.commandsToRegister).toHaveLength(100); + expect(result.totalCommands).toBe(105); + expect(result.maxCommands).toBe(100); + expect(result.overflowCount).toBe(5); + expect(result.commandsToRegister[0]).toEqual({ command: "cmd_0", description: "Command 0" }); + expect(result.commandsToRegister[99]).toEqual({ + command: "cmd_99", + description: "Command 99", + }); + }); + + it("validates plugin command specs and reports conflicts", () => { + const existingCommands = new Set(["native"]); + + const result = buildPluginTelegramMenuCommands({ + specs: [ + { name: "valid", description: " Works " }, + { name: "bad-name!", description: "Bad" }, + { name: "native", description: "Conflicts with native" }, + { name: "valid", description: "Duplicate plugin name" }, + { name: "empty", description: " " }, + ], + existingCommands, + }); + + expect(result.commands).toEqual([{ command: "valid", description: "Works" }]); + expect(result.issues).toContain( + 'Plugin command "/bad-name!" is invalid for Telegram (use a-z, 0-9, underscore; max 32 chars).', + ); + expect(result.issues).toContain( + 'Plugin command "/native" conflicts with an existing Telegram command.', + ); + expect(result.issues).toContain('Plugin command "/valid" is duplicated.'); + expect(result.issues).toContain('Plugin command "/empty" is missing a description.'); + }); + + it("deletes stale commands before setting new menu", async () => { + const callOrder: string[] = []; + const deleteMyCommands = vi.fn(async () => { + callOrder.push("delete"); + }); + const setMyCommands = vi.fn(async () => { + callOrder.push("set"); + }); + + syncTelegramMenuCommands({ + bot: { + api: { + deleteMyCommands, + setMyCommands, + }, + } as unknown as Parameters[0]["bot"], + runtime: {} as Parameters[0]["runtime"], + commandsToRegister: [{ command: "cmd", description: "Command" }], + }); + + await vi.waitFor(() => { + expect(setMyCommands).toHaveBeenCalled(); + }); + + expect(callOrder).toEqual(["delete", "set"]); + }); +}); diff --git a/src/telegram/bot-native-command-menu.ts b/src/telegram/bot-native-command-menu.ts new file mode 100644 index 00000000000..98fcf1e0d3b --- /dev/null +++ b/src/telegram/bot-native-command-menu.ts @@ -0,0 +1,104 @@ +import type { Bot } from "grammy"; +import type { RuntimeEnv } from "../runtime.js"; +import { + normalizeTelegramCommandName, + TELEGRAM_COMMAND_NAME_PATTERN, +} from "../config/telegram-custom-commands.js"; +import { withTelegramApiErrorLogging } from "./api-logging.js"; + +export const TELEGRAM_MAX_COMMANDS = 100; + +export type TelegramMenuCommand = { + command: string; + description: string; +}; + +type TelegramPluginCommandSpec = { + name: string; + description: string; +}; + +export function buildPluginTelegramMenuCommands(params: { + specs: TelegramPluginCommandSpec[]; + existingCommands: Set; +}): { commands: TelegramMenuCommand[]; issues: string[] } { + const { specs, existingCommands } = params; + const commands: TelegramMenuCommand[] = []; + const issues: string[] = []; + const pluginCommandNames = new Set(); + + for (const spec of specs) { + const normalized = normalizeTelegramCommandName(spec.name); + if (!normalized || !TELEGRAM_COMMAND_NAME_PATTERN.test(normalized)) { + issues.push( + `Plugin command "/${spec.name}" is invalid for Telegram (use a-z, 0-9, underscore; max 32 chars).`, + ); + continue; + } + const description = spec.description.trim(); + if (!description) { + issues.push(`Plugin command "/${normalized}" is missing a description.`); + continue; + } + if (existingCommands.has(normalized)) { + if (pluginCommandNames.has(normalized)) { + issues.push(`Plugin command "/${normalized}" is duplicated.`); + } else { + issues.push(`Plugin command "/${normalized}" conflicts with an existing Telegram command.`); + } + continue; + } + pluginCommandNames.add(normalized); + existingCommands.add(normalized); + commands.push({ command: normalized, description }); + } + + return { commands, issues }; +} + +export function buildCappedTelegramMenuCommands(params: { + allCommands: TelegramMenuCommand[]; + maxCommands?: number; +}): { + commandsToRegister: TelegramMenuCommand[]; + totalCommands: number; + maxCommands: number; + overflowCount: number; +} { + const { allCommands } = params; + const maxCommands = params.maxCommands ?? TELEGRAM_MAX_COMMANDS; + const totalCommands = allCommands.length; + const overflowCount = Math.max(0, totalCommands - maxCommands); + const commandsToRegister = allCommands.slice(0, maxCommands); + return { commandsToRegister, totalCommands, maxCommands, overflowCount }; +} + +export function syncTelegramMenuCommands(params: { + bot: Bot; + runtime: RuntimeEnv; + commandsToRegister: TelegramMenuCommand[]; +}): void { + const { bot, runtime, commandsToRegister } = params; + const sync = async () => { + // Keep delete -> set ordering to avoid stale deletions racing after fresh registrations. + if (typeof bot.api.deleteMyCommands === "function") { + await withTelegramApiErrorLogging({ + operation: "deleteMyCommands", + runtime, + fn: () => bot.api.deleteMyCommands(), + }).catch(() => {}); + } + + if (commandsToRegister.length === 0) { + return; + } + + await withTelegramApiErrorLogging({ + operation: "setMyCommands", + runtime, + fn: () => bot.api.setMyCommands(commandsToRegister), + }); + }; + + void sync().catch(() => {}); +} diff --git a/src/telegram/bot-native-commands.plugin-auth.test.ts b/src/telegram/bot-native-commands.plugin-auth.test.ts index 7572279b5c2..cabd3338019 100644 --- a/src/telegram/bot-native-commands.plugin-auth.test.ts +++ b/src/telegram/bot-native-commands.plugin-auth.test.ts @@ -23,6 +23,67 @@ vi.mock("../pairing/pairing-store.js", () => ({ })); describe("registerTelegramNativeCommands (plugin auth)", () => { + it("caps menu registration at 100 while leaving hidden plugin handlers available", () => { + const specs = Array.from({ length: 101 }, (_, i) => ({ + name: `cmd_${i}`, + description: `Command ${i}`, + })); + getPluginCommandSpecs.mockReturnValue(specs); + matchPluginCommand.mockReset(); + executePluginCommand.mockReset(); + deliverReplies.mockReset(); + + const handlers: Record Promise> = {}; + const setMyCommands = vi.fn().mockResolvedValue(undefined); + const log = vi.fn(); + const bot = { + api: { + setMyCommands, + sendMessage: vi.fn(), + }, + command: (name: string, handler: (ctx: unknown) => Promise) => { + handlers[name] = handler; + }, + } as const; + + registerTelegramNativeCommands({ + bot: bot as unknown as Parameters[0]["bot"], + cfg: {} as OpenClawConfig, + runtime: { log } as RuntimeEnv, + accountId: "default", + telegramCfg: {} as TelegramAccountConfig, + allowFrom: [], + groupAllowFrom: [], + replyToMode: "off", + textLimit: 4000, + useAccessGroups: false, + nativeEnabled: false, + nativeSkillsEnabled: false, + nativeDisabledExplicit: false, + resolveGroupPolicy: () => + ({ + allowlistEnabled: false, + allowed: true, + }) as ChannelGroupPolicy, + resolveTelegramGroupConfig: () => ({ + groupConfig: undefined, + topicConfig: undefined, + }), + shouldSkipUpdate: () => false, + opts: { token: "token" }, + }); + + const registered = setMyCommands.mock.calls[0]?.[0] as Array<{ + command: string; + description: string; + }>; + expect(registered).toHaveLength(100); + expect(registered[0]).toEqual({ command: "cmd_0", description: "Command 0" }); + expect(registered[99]).toEqual({ command: "cmd_99", description: "Command 99" }); + expect(log).toHaveBeenCalledWith(expect.stringContaining("registering first 100")); + expect(Object.keys(handlers)).toHaveLength(101); + }); + it("allows requireAuth:false plugin command even when sender is unauthorized", async () => { const command = { name: "plugin", diff --git a/src/telegram/bot-native-commands.test.ts b/src/telegram/bot-native-commands.test.ts index 48594c1e262..4f1f6f30781 100644 --- a/src/telegram/bot-native-commands.test.ts +++ b/src/telegram/bot-native-commands.test.ts @@ -67,7 +67,7 @@ describe("registerTelegramNativeCommands", () => { }); }); - it("keeps skill commands unscoped without a matching binding", () => { + it("scopes skill commands to default agent without a matching binding (#15599)", () => { const cfg: OpenClawConfig = { agents: { list: [{ id: "main", default: true }, { id: "butler" }], @@ -76,7 +76,10 @@ describe("registerTelegramNativeCommands", () => { registerTelegramNativeCommands(buildParams(cfg, "bot-a")); - expect(listSkillCommandsForAgents).toHaveBeenCalledWith({ cfg }); + expect(listSkillCommandsForAgents).toHaveBeenCalledWith({ + cfg, + agentIds: ["main"], + }); }); it("truncates Telegram command registration to 100 commands", () => { @@ -112,7 +115,7 @@ describe("registerTelegramNativeCommands", () => { expect(registeredCommands).toHaveLength(100); expect(registeredCommands).toEqual(customCommands.slice(0, 100)); expect(runtimeLog).toHaveBeenCalledWith( - "telegram: truncating 120 commands to 100 (Telegram Bot API limit)", + "Telegram limits bots to 100 commands. 120 configured; registering first 100. Use channels.telegram.commands.native: false to disable, or reduce plugin/skill/custom commands.", ); }); }); diff --git a/src/telegram/bot-native-commands.ts b/src/telegram/bot-native-commands.ts index 3983af3691b..b15a761ffd9 100644 --- a/src/telegram/bot-native-commands.ts +++ b/src/telegram/bot-native-commands.ts @@ -26,10 +26,6 @@ import { resolveCommandAuthorizedFromAuthorizers } from "../channels/command-gat import { createReplyPrefixOptions } from "../channels/reply-prefix.js"; import { resolveMarkdownTableMode } from "../config/markdown-tables.js"; import { resolveTelegramCustomCommands } from "../config/telegram-custom-commands.js"; -import { - normalizeTelegramCommandName, - TELEGRAM_COMMAND_NAME_PATTERN, -} from "../config/telegram-custom-commands.js"; import { danger, logVerbose } from "../globals.js"; import { getChildLogger } from "../logging.js"; import { readChannelAllowFromStore } from "../pairing/pairing-store.js"; @@ -42,6 +38,11 @@ import { resolveAgentRoute } from "../routing/resolve-route.js"; import { resolveThreadSessionKeys } from "../routing/session-key.js"; import { withTelegramApiErrorLogging } from "./api-logging.js"; import { firstDefined, isSenderAllowed, normalizeAllowFromWithStore } from "./bot-access.js"; +import { + buildCappedTelegramMenuCommands, + buildPluginTelegramMenuCommands, + syncTelegramMenuCommands, +} from "./bot-native-command-menu.js"; import { TelegramUpdateKeyContext } from "./bot-updates.js"; import { TelegramBotOptions } from "./bot.js"; import { deliverReplies } from "./bot/delivery.js"; @@ -294,8 +295,7 @@ export const registerTelegramNativeCommands = ({ nativeEnabled && nativeSkillsEnabled ? resolveAgentRoute({ cfg, channel: "telegram", accountId }) : null; - const boundAgentIds = - boundRoute && boundRoute.matchedBy.startsWith("binding.") ? [boundRoute.agentId] : null; + const boundAgentIds = boundRoute ? [boundRoute.agentId] : null; const skillCommands = nativeEnabled && nativeSkillsEnabled ? listSkillCommandsForAgents(boundAgentIds ? { cfg, agentIds: boundAgentIds } : { cfg }) @@ -321,87 +321,43 @@ export const registerTelegramNativeCommands = ({ } const customCommands = customResolution.commands; const pluginCommandSpecs = getPluginCommandSpecs(); - const pluginCommands: Array<{ command: string; description: string }> = []; const existingCommands = new Set( [ ...nativeCommands.map((command) => command.name), ...customCommands.map((command) => command.command), ].map((command) => command.toLowerCase()), ); - const pluginCommandNames = new Set(); - for (const spec of pluginCommandSpecs) { - const normalized = normalizeTelegramCommandName(spec.name); - if (!normalized || !TELEGRAM_COMMAND_NAME_PATTERN.test(normalized)) { - runtime.error?.( - danger( - `Plugin command "/${spec.name}" is invalid for Telegram (use a-z, 0-9, underscore; max 32 chars).`, - ), - ); - continue; - } - const description = spec.description.trim(); - if (!description) { - runtime.error?.(danger(`Plugin command "/${normalized}" is missing a description.`)); - continue; - } - if (existingCommands.has(normalized)) { - runtime.error?.( - danger(`Plugin command "/${normalized}" conflicts with an existing Telegram command.`), - ); - continue; - } - if (pluginCommandNames.has(normalized)) { - runtime.error?.(danger(`Plugin command "/${normalized}" is duplicated.`)); - continue; - } - pluginCommandNames.add(normalized); - existingCommands.add(normalized); - pluginCommands.push({ command: normalized, description }); + const pluginCatalog = buildPluginTelegramMenuCommands({ + specs: pluginCommandSpecs, + existingCommands, + }); + for (const issue of pluginCatalog.issues) { + runtime.error?.(danger(issue)); } const allCommandsFull: Array<{ command: string; description: string }> = [ ...nativeCommands.map((command) => ({ command: command.name, description: command.description, })), - ...pluginCommands, + ...pluginCatalog.commands, ...customCommands, ]; - // Telegram Bot API limits commands to 100 per scope. - // Truncate with a warning rather than failing with BOT_COMMANDS_TOO_MUCH. - const TELEGRAM_MAX_COMMANDS = 100; - if (allCommandsFull.length > TELEGRAM_MAX_COMMANDS) { + const { commandsToRegister, totalCommands, maxCommands, overflowCount } = + buildCappedTelegramMenuCommands({ + allCommands: allCommandsFull, + }); + if (overflowCount > 0) { runtime.log?.( - `telegram: truncating ${allCommandsFull.length} commands to ${TELEGRAM_MAX_COMMANDS} (Telegram Bot API limit)`, + `Telegram limits bots to ${maxCommands} commands. ` + + `${totalCommands} configured; registering first ${maxCommands}. ` + + `Use channels.telegram.commands.native: false to disable, or reduce plugin/skill/custom commands.`, ); } - const allCommands = allCommandsFull.slice(0, TELEGRAM_MAX_COMMANDS); + // Telegram only limits the setMyCommands payload (menu entries). + // Keep hidden commands callable by registering handlers for the full catalog. + syncTelegramMenuCommands({ bot, runtime, commandsToRegister }); - // Clear stale commands before registering new ones to prevent - // leftover commands from deleted skills persisting across restarts (#5717). - // Chain delete β†’ set so a late-resolving delete cannot wipe newly registered commands. - const registerCommands = () => { - if (allCommands.length > 0) { - withTelegramApiErrorLogging({ - operation: "setMyCommands", - runtime, - fn: () => bot.api.setMyCommands(allCommands), - }).catch(() => {}); - } - }; - if (typeof bot.api.deleteMyCommands === "function") { - withTelegramApiErrorLogging({ - operation: "deleteMyCommands", - runtime, - fn: () => bot.api.deleteMyCommands(), - }) - .catch(() => {}) - .then(registerCommands) - .catch(() => {}); - } else { - registerCommands(); - } - - if (allCommands.length > 0) { + if (commandsToRegister.length > 0) { if (typeof (bot as unknown as { command?: unknown }).command !== "function") { logVerbose("telegram: bot.command unavailable; skipping native handlers"); } else { @@ -642,7 +598,7 @@ export const registerTelegramNativeCommands = ({ }); } - for (const pluginCommand of pluginCommands) { + for (const pluginCommand of pluginCatalog.commands) { bot.command(pluginCommand.command, async (ctx: TelegramNativeCommandContext) => { const msg = ctx.message; if (!msg) { diff --git a/src/telegram/bot.media.downloads-media-file-path-no-file-download.e2e.test.ts b/src/telegram/bot.media.downloads-media-file-path-no-file-download.e2e.test.ts index e0440b3a313..b85c8c018c0 100644 --- a/src/telegram/bot.media.downloads-media-file-path-no-file-download.e2e.test.ts +++ b/src/telegram/bot.media.downloads-media-file-path-no-file-download.e2e.test.ts @@ -14,8 +14,8 @@ const resolvePinnedHostname = ssrf.resolvePinnedHostname; const lookupMock = vi.fn(); let resolvePinnedHostnameSpy: ReturnType = null; const TELEGRAM_TEST_TIMINGS = { - mediaGroupFlushMs: 75, - textFragmentGapMs: 120, + mediaGroupFlushMs: 20, + textFragmentGapMs: 30, } as const; const sleep = async (ms: number) => { @@ -300,7 +300,7 @@ describe("telegram media groups", () => { }); const MEDIA_GROUP_TEST_TIMEOUT_MS = process.platform === "win32" ? 45_000 : 20_000; - const MEDIA_GROUP_FLUSH_MS = TELEGRAM_TEST_TIMINGS.mediaGroupFlushMs + 120; + const MEDIA_GROUP_FLUSH_MS = TELEGRAM_TEST_TIMINGS.mediaGroupFlushMs + 60; it( "buffers messages with same media_group_id and processes them together", @@ -737,7 +737,7 @@ describe("telegram text fragments", () => { }); const TEXT_FRAGMENT_TEST_TIMEOUT_MS = process.platform === "win32" ? 45_000 : 20_000; - const TEXT_FRAGMENT_FLUSH_MS = TELEGRAM_TEST_TIMINGS.textFragmentGapMs + 160; + const TEXT_FRAGMENT_FLUSH_MS = TELEGRAM_TEST_TIMINGS.textFragmentGapMs + 80; it( "buffers near-limit text and processes sequential parts as one message", diff --git a/src/telegram/bot.test.ts b/src/telegram/bot.test.ts index 3c2c63a7d40..cb919a0237f 100644 --- a/src/telegram/bot.test.ts +++ b/src/telegram/bot.test.ts @@ -1,7 +1,7 @@ import fs from "node:fs"; import os from "node:os"; import path from "node:path"; -import { afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; import { escapeRegExp, formatEnvelopeTimestamp } from "../../test/helpers/envelope-timestamp.js"; import { expectInboundContextContract } from "../../test/helpers/inbound-contract.js"; import { @@ -23,6 +23,13 @@ vi.mock("../auto-reply/skill-commands.js", () => ({ const { sessionStorePath } = vi.hoisted(() => ({ sessionStorePath: `/tmp/openclaw-telegram-bot-${Math.random().toString(16).slice(2)}.json`, })); +const tempDirs: string[] = []; + +function createTempDir(prefix: string): string { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), prefix)); + tempDirs.push(dir); + return dir; +} function resolveSkillCommands(config: Parameters[0]) { return listSkillCommandsForAgents({ cfg: config }); @@ -208,6 +215,13 @@ describe("createTelegramBot", () => { process.env.TZ = ORIGINAL_TZ; }); + afterAll(() => { + for (const dir of tempDirs) { + fs.rmSync(dir, { recursive: true, force: true }); + } + tempDirs.length = 0; + }); + it("installs grammY throttler", () => { createTelegramBot({ token: "tok" }); expect(throttlerSpy).toHaveBeenCalledTimes(1); @@ -1214,7 +1228,7 @@ describe("createTelegramBot", () => { onSpy.mockReset(); const replySpy = replyModule.__replySpy as unknown as ReturnType; replySpy.mockReset(); - const storeDir = fs.mkdtempSync(path.join(os.tmpdir(), "openclaw-telegram-")); + const storeDir = createTempDir("openclaw-telegram-"); const storePath = path.join(storeDir, "sessions.json"); fs.writeFileSync( storePath, diff --git a/src/telegram/bot/delivery.ts b/src/telegram/bot/delivery.ts index bd97d570889..732227ed023 100644 --- a/src/telegram/bot/delivery.ts +++ b/src/telegram/bot/delivery.ts @@ -18,6 +18,7 @@ import { markdownToTelegramChunks, markdownToTelegramHtml, renderTelegramHtmlText, + wrapFileReferencesInHtml, } from "../format.js"; import { buildInlineKeyboard } from "../send.js"; import { cacheSticker, getCachedSticker } from "../sticker-cache.js"; @@ -76,7 +77,9 @@ export async function deliverReplies(params: { const nested = markdownToTelegramChunks(chunk, textLimit, { tableMode: params.tableMode }); if (!nested.length && chunk) { chunks.push({ - html: markdownToTelegramHtml(chunk, { tableMode: params.tableMode }), + html: wrapFileReferencesInHtml( + markdownToTelegramHtml(chunk, { tableMode: params.tableMode, wrapFileRefs: false }), + ), text: chunk, }); continue; diff --git a/src/telegram/format.test.ts b/src/telegram/format.test.ts index 48e95343750..6b0e1944f70 100644 --- a/src/telegram/format.test.ts +++ b/src/telegram/format.test.ts @@ -95,6 +95,18 @@ describe("markdownToTelegramHtml", () => { expect(res).toBe('bold'); }); + it("wraps punctuated file references in code tags", () => { + const res = markdownToTelegramHtml("See README.md. Also (backup.sh)."); + expect(res).toContain("README.md."); + expect(res).toContain("(backup.sh)."); + }); + + it("keeps .co domains as links", () => { + const res = markdownToTelegramHtml("Visit t.co and openclaw.co"); + expect(res).toContain('t.co'); + expect(res).toContain('openclaw.co'); + }); + it("renders spoiler tags", () => { const res = markdownToTelegramHtml("the answer is ||42||"); expect(res).toBe("the answer is 42"); diff --git a/src/telegram/format.ts b/src/telegram/format.ts index eb457edff0c..f919a917f9f 100644 --- a/src/telegram/format.ts +++ b/src/telegram/format.ts @@ -20,7 +20,56 @@ function escapeHtmlAttr(text: string): string { return escapeHtml(text).replace(/"/g, """); } -function buildTelegramLink(link: MarkdownLinkSpan, _text: string) { +/** + * File extensions that share TLDs and commonly appear in code/documentation. + * These are wrapped in tags to prevent Telegram from generating + * spurious domain registrar previews. + * + * Only includes extensions that are: + * 1. Commonly used as file extensions in code/docs + * 2. Rarely used as intentional domain references + * + * Excluded: .ai, .io, .tv, .fm (popular domain TLDs like x.ai, vercel.io, github.io) + */ +const FILE_EXTENSIONS_WITH_TLD = new Set([ + "md", // Markdown (Moldova) - very common in repos + "go", // Go language - common in Go projects + "py", // Python (Paraguay) - common in Python projects + "pl", // Perl (Poland) - common in Perl projects + "sh", // Shell (Saint Helena) - common for scripts + "am", // Automake files (Armenia) + "at", // Assembly (Austria) + "be", // Backend files (Belgium) + "cc", // C++ source (Cocos Islands) +]); + +/** Detects when markdown-it linkify auto-generated a link from a bare filename (e.g. README.md β†’ http://README.md) */ +function isAutoLinkedFileRef(href: string, label: string): boolean { + const stripped = href.replace(/^https?:\/\//i, ""); + if (stripped !== label) { + return false; + } + const dotIndex = label.lastIndexOf("."); + if (dotIndex < 1) { + return false; + } + const ext = label.slice(dotIndex + 1).toLowerCase(); + if (!FILE_EXTENSIONS_WITH_TLD.has(ext)) { + return false; + } + // Reject if any path segment before the filename contains a dot (looks like a domain) + const segments = label.split("/"); + if (segments.length > 1) { + for (let i = 0; i < segments.length - 1; i++) { + if (segments[i].includes(".")) { + return false; + } + } + } + return true; +} + +function buildTelegramLink(link: MarkdownLinkSpan, text: string) { const href = link.href.trim(); if (!href) { return null; @@ -28,6 +77,11 @@ function buildTelegramLink(link: MarkdownLinkSpan, _text: string) { if (link.start === link.end) { return null; } + // Suppress auto-linkified file references (e.g. README.md β†’ http://README.md) + const label = text.slice(link.start, link.end); + if (isAutoLinkedFileRef(href, label)) { + return null; + } const safeHref = escapeHtmlAttr(href); return { start: link.start, @@ -55,7 +109,7 @@ function renderTelegramHtml(ir: MarkdownIR): string { export function markdownToTelegramHtml( markdown: string, - options: { tableMode?: MarkdownTableMode } = {}, + options: { tableMode?: MarkdownTableMode; wrapFileRefs?: boolean } = {}, ): string { const ir = markdownToIR(markdown ?? "", { linkify: true, @@ -64,7 +118,114 @@ export function markdownToTelegramHtml( blockquotePrefix: "", tableMode: options.tableMode, }); - return renderTelegramHtml(ir); + const html = renderTelegramHtml(ir); + // Apply file reference wrapping if requested (for chunked rendering) + if (options.wrapFileRefs !== false) { + return wrapFileReferencesInHtml(html); + } + return html; +} + +/** + * Wraps standalone file references (with TLD extensions) in tags. + * This prevents Telegram from treating them as URLs and generating + * irrelevant domain registrar previews. + * + * Runs AFTER markdownβ†’HTML conversion to avoid modifying HTML attributes. + * Skips content inside , 
, and  tags to avoid nesting issues.
+ */
+/** Escape regex metacharacters in a string */
+function escapeRegex(str: string): string {
+  return str.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
+}
+
+const FILE_EXTENSIONS_PATTERN = Array.from(FILE_EXTENSIONS_WITH_TLD).map(escapeRegex).join("|");
+const AUTO_LINKED_ANCHOR_PATTERN = /]*>\1<\/a>/gi;
+const FILE_REFERENCE_PATTERN = new RegExp(
+  `(^|[^a-zA-Z0-9_\\-/])([a-zA-Z0-9_.\\-./]+\\.(?:${FILE_EXTENSIONS_PATTERN}))(?=$|[^a-zA-Z0-9_\\-/])`,
+  "gi",
+);
+const ORPHANED_TLD_PATTERN = new RegExp(
+  `([^a-zA-Z0-9]|^)([A-Za-z]\\.(?:${FILE_EXTENSIONS_PATTERN}))(?=[^a-zA-Z0-9/]|$)`,
+  "g",
+);
+const HTML_TAG_PATTERN = /(<\/?)([a-zA-Z][a-zA-Z0-9-]*)\b[^>]*?>/gi;
+
+function wrapStandaloneFileRef(match: string, prefix: string, filename: string): string {
+  if (filename.startsWith("//")) {
+    return match;
+  }
+  if (/https?:\/\/$/i.test(prefix)) {
+    return match;
+  }
+  return `${prefix}${escapeHtml(filename)}`;
+}
+
+function wrapSegmentFileRefs(
+  text: string,
+  codeDepth: number,
+  preDepth: number,
+  anchorDepth: number,
+): string {
+  if (!text || codeDepth > 0 || preDepth > 0 || anchorDepth > 0) {
+    return text;
+  }
+  const wrappedStandalone = text.replace(FILE_REFERENCE_PATTERN, wrapStandaloneFileRef);
+  return wrappedStandalone.replace(ORPHANED_TLD_PATTERN, (match, prefix: string, tld: string) =>
+    prefix === ">" ? match : `${prefix}${escapeHtml(tld)}`,
+  );
+}
+
+export function wrapFileReferencesInHtml(html: string): string {
+  // Safety-net: de-linkify auto-generated anchors where href="http://Link';
+    const result = wrapFileReferencesInHtml(input);
+    expect(result).toBe(input);
+  });
+
+  it("does not wrap file refs inside real URL anchor tags", () => {
+    const input = 'Visit example.com/README.md';
+    const result = wrapFileReferencesInHtml(input);
+    expect(result).toBe(input);
+  });
+
+  it("handles mixed content correctly", () => {
+    const result = wrapFileReferencesInHtml("Check README.md and CONTRIBUTING.md");
+    expect(result).toContain("README.md");
+    expect(result).toContain("CONTRIBUTING.md");
+  });
+
+  it("handles edge cases", () => {
+    expect(wrapFileReferencesInHtml("No markdown files here")).not.toContain("");
+    expect(wrapFileReferencesInHtml("File.md at start")).toContain("File.md");
+    expect(wrapFileReferencesInHtml("Ends with file.md")).toContain("file.md");
+  });
+
+  it("wraps file refs with punctuation boundaries", () => {
+    expect(wrapFileReferencesInHtml("See README.md.")).toContain("README.md.");
+    expect(wrapFileReferencesInHtml("See README.md,")).toContain("README.md,");
+    expect(wrapFileReferencesInHtml("(README.md)")).toContain("(README.md)");
+    expect(wrapFileReferencesInHtml("README.md:")).toContain("README.md:");
+  });
+
+  it("de-linkifies auto-linkified file ref anchors", () => {
+    const input = 'README.md';
+    expect(wrapFileReferencesInHtml(input)).toBe("README.md");
+  });
+
+  it("de-linkifies auto-linkified path anchors", () => {
+    const input = 'squad/friday/HEARTBEAT.md';
+    expect(wrapFileReferencesInHtml(input)).toBe("squad/friday/HEARTBEAT.md");
+  });
+
+  it("preserves explicit links where label differs from href", () => {
+    const input = 'click here';
+    expect(wrapFileReferencesInHtml(input)).toBe(input);
+  });
+
+  it("wraps file ref after closing anchor tag", () => {
+    const input = 'link then README.md';
+    const result = wrapFileReferencesInHtml(input);
+    expect(result).toContain(" then README.md");
+  });
+});
+
+describe("renderTelegramHtmlText - file reference wrapping", () => {
+  it("wraps file references in markdown mode", () => {
+    const result = renderTelegramHtmlText("Check README.md");
+    expect(result).toContain("README.md");
+  });
+
+  it("does not wrap in HTML mode (trusts caller markup)", () => {
+    // textMode: "html" should pass through unchanged - caller owns the markup
+    const result = renderTelegramHtmlText("Check README.md", { textMode: "html" });
+    expect(result).toBe("Check README.md");
+    expect(result).not.toContain("");
+  });
+
+  it("does not double-wrap already code-formatted content", () => {
+    const result = renderTelegramHtmlText("Already `wrapped.md` here");
+    // Should have code tags but not nested
+    expect(result).toContain("");
+    expect(result).not.toContain("");
+  });
+});
+
+describe("markdownToTelegramHtml - file reference wrapping", () => {
+  it("wraps file references by default", () => {
+    const result = markdownToTelegramHtml("Check README.md");
+    expect(result).toContain("README.md");
+  });
+
+  it("can skip wrapping when requested", () => {
+    const result = markdownToTelegramHtml("Check README.md", { wrapFileRefs: false });
+    expect(result).not.toContain("README.md");
+  });
+
+  it("wraps multiple file types in a single message", () => {
+    const result = markdownToTelegramHtml("Edit main.go and script.py");
+    expect(result).toContain("main.go");
+    expect(result).toContain("script.py");
+  });
+
+  it("preserves real URLs as anchor tags", () => {
+    const result = markdownToTelegramHtml("Visit https://example.com");
+    expect(result).toContain('');
+  });
+
+  it("preserves explicit markdown links even when href looks like a file ref", () => {
+    const result = markdownToTelegramHtml("[docs](http://README.md)");
+    expect(result).toContain('docs');
+  });
+
+  it("wraps file ref after real URL in same message", () => {
+    const result = markdownToTelegramHtml("Visit https://example.com and README.md");
+    expect(result).toContain('');
+    expect(result).toContain("README.md");
+  });
+});
+
+describe("markdownToTelegramChunks - file reference wrapping", () => {
+  it("wraps file references in chunked output", () => {
+    const chunks = markdownToTelegramChunks("Check README.md and backup.sh", 4096);
+    expect(chunks.length).toBeGreaterThan(0);
+    expect(chunks[0].html).toContain("README.md");
+    expect(chunks[0].html).toContain("backup.sh");
+  });
+});
+
+describe("edge cases", () => {
+  it("wraps file ref inside bold tags", () => {
+    const result = markdownToTelegramHtml("**README.md**");
+    expect(result).toBe("README.md");
+  });
+
+  it("wraps file ref inside italic tags", () => {
+    const result = markdownToTelegramHtml("*script.py*");
+    expect(result).toBe("script.py");
+  });
+
+  it("does not wrap inside fenced code blocks", () => {
+    const result = markdownToTelegramHtml("```\nREADME.md\n```");
+    expect(result).toBe("
README.md\n
");
+    expect(result).not.toContain("");
+  });
+
+  it("preserves domain-like paths as anchor tags", () => {
+    const result = markdownToTelegramHtml("example.com/README.md");
+    expect(result).toContain('');
+    expect(result).not.toContain("");
+  });
+
+  it("preserves github URLs with file paths", () => {
+    const result = markdownToTelegramHtml("https://github.com/foo/README.md");
+    expect(result).toContain('');
+  });
+
+  it("handles wrapFileRefs: false (plain text output)", () => {
+    const result = markdownToTelegramHtml("README.md", { wrapFileRefs: false });
+    // buildTelegramLink returns null, so no  tag; wrapFileRefs: false skips 
+    expect(result).toBe("README.md");
+  });
+
+  it("wraps supported TLD extensions (.am, .at, .be, .cc)", () => {
+    const result = markdownToTelegramHtml("Makefile.am and code.at and app.be and main.cc");
+    expect(result).toContain("Makefile.am");
+    expect(result).toContain("code.at");
+    expect(result).toContain("app.be");
+    expect(result).toContain("main.cc");
+  });
+
+  it("does not wrap popular domain TLDs (.ai, .io, .tv, .fm)", () => {
+    // These are commonly used as real domains (x.ai, vercel.io, github.io)
+    const result = markdownToTelegramHtml("Check x.ai and vercel.io and app.tv and radio.fm");
+    // Should be links, not code
+    expect(result).toContain('');
+    expect(result).toContain('');
+    expect(result).toContain('');
+    expect(result).toContain('');
+  });
+
+  it("keeps .co domains as links", () => {
+    const result = markdownToTelegramHtml("Visit t.co and openclaw.co");
+    expect(result).toContain('');
+    expect(result).toContain('');
+    expect(result).not.toContain("t.co");
+    expect(result).not.toContain("openclaw.co");
+  });
+
+  it("does not wrap non-TLD extensions", () => {
+    const result = markdownToTelegramHtml("image.png and style.css and script.js");
+    expect(result).not.toContain("image.png");
+    expect(result).not.toContain("style.css");
+    expect(result).not.toContain("script.js");
+  });
+
+  it("handles file ref at start of message", () => {
+    const result = markdownToTelegramHtml("README.md is important");
+    expect(result).toBe("README.md is important");
+  });
+
+  it("handles file ref at end of message", () => {
+    const result = markdownToTelegramHtml("Check the README.md");
+    expect(result).toBe("Check the README.md");
+  });
+
+  it("handles multiple file refs in sequence", () => {
+    const result = markdownToTelegramHtml("README.md CHANGELOG.md LICENSE.md");
+    expect(result).toContain("README.md");
+    expect(result).toContain("CHANGELOG.md");
+    expect(result).toContain("LICENSE.md");
+  });
+
+  it("handles nested path without domain-like segments", () => {
+    const result = markdownToTelegramHtml("src/utils/helpers/format.go");
+    expect(result).toContain("src/utils/helpers/format.go");
+  });
+
+  it("wraps path with version-like segment (not a domain)", () => {
+    // v1.0/README.md is not linkified by markdown-it (no TLD), so it's wrapped
+    const result = markdownToTelegramHtml("v1.0/README.md");
+    expect(result).toContain("v1.0/README.md");
+  });
+
+  it("preserves domain path with version segment", () => {
+    // example.com/v1.0/README.md IS linkified (has domain), preserved as link
+    const result = markdownToTelegramHtml("example.com/v1.0/README.md");
+    expect(result).toContain('');
+  });
+
+  it("handles file ref with hyphen and underscore in name", () => {
+    const result = markdownToTelegramHtml("my-file_name.md");
+    expect(result).toContain("my-file_name.md");
+  });
+
+  it("handles uppercase extensions", () => {
+    const result = markdownToTelegramHtml("README.MD and SCRIPT.PY");
+    expect(result).toContain("README.MD");
+    expect(result).toContain("SCRIPT.PY");
+  });
+
+  it("handles nested code tags (depth tracking)", () => {
+    // Nested  inside 
 - should not wrap inner content
+    const input = "
README.md
 then script.py";
+    const result = wrapFileReferencesInHtml(input);
+    expect(result).toBe("
README.md
 then script.py");
+  });
+
+  it("handles multiple anchor tags in sequence", () => {
+    const input =
+      'link1 README.md link2 script.py';
+    const result = wrapFileReferencesInHtml(input);
+    expect(result).toContain(" README.md  script.py");
+  });
+
+  it("handles auto-linked anchor with backreference match", () => {
+    // The regex uses \1 backreference - href must equal label
+    const input = 'README.md';
+    expect(wrapFileReferencesInHtml(input)).toBe("README.md");
+  });
+
+  it("preserves anchor when href and label differ (no backreference match)", () => {
+    // Different href and label - should NOT de-linkify
+    const input = 'README.md';
+    expect(wrapFileReferencesInHtml(input)).toBe(input);
+  });
+
+  it("wraps orphaned TLD pattern after special character", () => {
+    // R&D.md - the & breaks the main pattern, but D.md could be auto-linked
+    // So we wrap the orphaned D.md part to prevent Telegram linking it
+    const input = "R&D.md";
+    const result = wrapFileReferencesInHtml(input);
+    expect(result).toBe("R&D.md");
+  });
+
+  it("wraps orphaned single-letter TLD patterns", () => {
+    // Use extensions still in the set (md, sh, py, go)
+    const result1 = wrapFileReferencesInHtml("X.md is cool");
+    expect(result1).toContain("X.md");
+
+    const result2 = wrapFileReferencesInHtml("Check R.sh");
+    expect(result2).toContain("R.sh");
+  });
+
+  it("does not match filenames containing angle brackets", () => {
+    // The regex character class [a-zA-Z0-9_.\\-./] doesn't include < >
+    // so these won't be matched and wrapped (which is correct/safe)
+    const input = "file