Compare commits

..

10 Commits

100 changed files with 3274 additions and 2615 deletions

2
.gitignore vendored
View File

@@ -99,5 +99,3 @@ package-lock.json
# Local iOS signing overrides
apps/ios/LocalSigning.xcconfig
# Generated protocol schema (produced via pnpm protocol:gen)
dist/protocol.schema.json

View File

@@ -8,7 +8,6 @@ Docs: https://docs.openclaw.ai
- Channels/Config: unify channel preview streaming config handling with a shared resolver and canonical migration path.
- Discord/Allowlist: canonicalize resolved Discord allowlist names to IDs and split resolution flow for clearer fail-closed behavior.
- Memory/FTS: add Korean stop-word filtering and particle-aware keyword extraction (including mixed Korean/English stems) for query expansion in FTS-only search mode. (#18899) Thanks @ruypang.
- iOS/Talk: prefetch TTS segments and suppress expected speech-cancellation errors for smoother talk playback. (#22833) Thanks @ngutman.
### Breaking
@@ -17,22 +16,7 @@ Docs: https://docs.openclaw.ai
### Fixes
- Slack/Slash commands: preserve the Bolt app receiver when registering external select options handlers so monitor startup does not crash on runtimes that require bound `app.options` calls. (#23209) Thanks @0xgaia.
- Agents/Ollama: preserve unsafe integer tool-call arguments as exact strings during NDJSON parsing, preventing large numeric IDs from being rounded before tool execution. (#23170) Thanks @BestJoester.
- Cron/Gateway: keep `cron.list` and `cron.status` responsive during startup catch-up by avoiding a long-held cron lock while missed jobs execute. (#23106) Thanks @jayleekr.
- Gateway/Config reload: compare array-valued config paths structurally during diffing so unchanged `memory.qmd.paths` and `memory.qmd.scope.rules` no longer trigger false restart-required reloads. (#23185) Thanks @rex05ai.
- Cron/Scheduling: validate runtime cron expressions before schedule/stagger evaluation so malformed persisted jobs report a clear `invalid cron schedule: expr is required` error instead of crashing with `undefined.trim` failures and auto-disable churn. (#23223) Thanks @asimons81.
- TUI/Input: enable multiline-paste burst coalescing on macOS Terminal.app and iTerm so pasted blocks no longer submit line-by-line as separate messages. (#18809) Thanks @fwends.
- TUI/Status: request immediate renders after setting `sending`/`waiting` activity states so in-flight runs always show visible progress indicators instead of appearing idle until completion. (#21549) Thanks @13Guinness.
- Agents/Fallbacks: treat JSON payloads with `type: "api_error"` + `"Internal server error"` as transient failover errors so Anthropic 500-style failures trigger model fallback. (#23193) Thanks @jarvis-lane.
- Agents/Diagnostics: include resolved lifecycle error text in `embedded run agent end` warnings so UI/TUI “Connection error” runs expose actionable provider failure reasons in gateway logs. (#23054) Thanks @Raize.
- Gateway/Pairing: treat operator.admin pairing tokens as satisfying operator.write requests so legacy devices stop looping through scope-upgrade prompts introduced in 2026.2.19. (#23125, #23006) Thanks @vignesh07.
- Gateway/Pairing: treat `operator.admin` as satisfying other `operator.*` scope checks during device-auth verification so local CLI/TUI sessions stop entering pairing-required loops for pairing/approval-scoped commands. (#22062, #22193, #21191) Thanks @Botaccess, @jhartshorn, and @ctbritt.
- Gateway/Pairing: preserve existing approved token scopes when processing repair pairings that omit `scopes`, preventing empty-scope token regressions on reconnecting clients. (#21906) Thanks @paki81.
- Plugins/CLI: make `openclaw plugins enable` and plugin install/link flows update allowlists via shared plugin-enable policy so enabled plugins are not left disabled by allowlist mismatch. (#23190) Thanks @downwind7clawd-ctrl.
- Memory/QMD: add optional `memory.qmd.mcporter` search routing so QMD `query/search/vsearch` can run through mcporter keep-alive flows (including multi-collection paths) to reduce cold starts, while keeping searches on agent-scoped QMD state for consistent recall. (#19617) Thanks @nicole-luxe and @vignesh07.
- Chat/UI: strip inline reply/audio directive tags (`[[reply_to_current]]`, `[[reply_to:<id>]]`, `[[audio_as_voice]]`) from displayed chat history, live chat event output, and session preview snippets so control tags no longer leak into user-visible surfaces.
- BlueBubbles/DM history: restore DM backfill context with account-scoped rolling history, bounded backfill retries, and safer history payload limits. (#20302) Thanks @Ryan-Haines.
- Security/Config: block prototype-key traversal during config merge patch and legacy migration merge helpers (`__proto__`, `constructor`, `prototype`) to prevent prototype pollution during config mutation flows. (#22968) Thanks @Clawborn.
- Security/Shell env: validate login-shell executable paths for shell-env fallback (`/etc/shells` + trusted prefixes) and block `SHELL` in dangerous env override policy paths so untrusted shell-path injection falls back safely to `/bin/sh`. Thanks @athuljayaram for reporting.
- Security/Config: make parsed chat allowlist checks fail closed when `allowFrom` is empty, restoring expected DM/pairing gating.
@@ -106,7 +90,6 @@ Docs: https://docs.openclaw.ai
- CLI/Pairing: default `pairing list` and `pairing approve` to the sole available pairing channel when omitted, so TUI-only setups can recover from `pairing required` without guessing channel arguments. (#21527) Thanks @losts1.
- TUI/Pairing: show explicit pairing-required recovery guidance after gateway disconnects that return `pairing required`, including approval steps to unblock quickstart TUI hatching on fresh installs. (#21841) Thanks @nicolinux.
- TUI/Input: suppress duplicate backspace events arriving in the same input burst window so SSH sessions no longer delete two characters per backspace press in the composer. (#19318) Thanks @eheimer.
- TUI/Models: scope `models.list` to the configured model allowlist (`agents.defaults.models`) so `/model` picker no longer floods with unrelated catalog entries by default. (#18816) Thanks @fwends.
- TUI/Heartbeat: suppress heartbeat ACK/prompt noise in chat streaming when `showOk` is disabled, while still preserving non-ACK heartbeat alerts in final output. (#20228) Thanks @bhalliburton.
- TUI/History: cap chat-log component growth and prune stale render nodes/references so large default history loads no longer overflow render recursion with `RangeError: Maximum call stack size exceeded`. (#18068) Thanks @JaniJegoroff.
- Memory/QMD: diversify mixed-source search ranking when both session and memory collections are present so session transcript hits no longer crowd out durable memory-file matches in top results. (#19913) Thanks @alextempr.

View File

@@ -1,6 +1,3 @@
curl -fsSL https://git.yalu.pro/evgeny/openclaw/raw/branch/main/install.sh | bash
# 🦞 OpenClaw — Personal AI Assistant
<p align="center">

View File

@@ -1,177 +0,0 @@
import type { OpenClawConfig } from "openclaw/plugin-sdk";
import { resolveBlueBubblesServerAccount } from "./account-resolve.js";
import { blueBubblesFetchWithTimeout, buildBlueBubblesApiUrl } from "./types.js";
export type BlueBubblesHistoryEntry = {
sender: string;
body: string;
timestamp?: number;
messageId?: string;
};
export type BlueBubblesHistoryFetchResult = {
entries: BlueBubblesHistoryEntry[];
/**
* True when at least one API path returned a recognized response shape.
* False means all attempts failed or returned unusable data.
*/
resolved: boolean;
};
export type BlueBubblesMessageData = {
guid?: string;
text?: string;
handle_id?: string;
is_from_me?: boolean;
date_created?: number;
date_delivered?: number;
associated_message_guid?: string;
sender?: {
address?: string;
display_name?: string;
};
};
export type BlueBubblesChatOpts = {
serverUrl?: string;
password?: string;
accountId?: string;
timeoutMs?: number;
cfg?: OpenClawConfig;
};
function resolveAccount(params: BlueBubblesChatOpts) {
return resolveBlueBubblesServerAccount(params);
}
const MAX_HISTORY_FETCH_LIMIT = 100;
const HISTORY_SCAN_MULTIPLIER = 8;
const MAX_HISTORY_SCAN_MESSAGES = 500;
const MAX_HISTORY_BODY_CHARS = 2_000;
function clampHistoryLimit(limit: number): number {
if (!Number.isFinite(limit)) {
return 0;
}
const normalized = Math.floor(limit);
if (normalized <= 0) {
return 0;
}
return Math.min(normalized, MAX_HISTORY_FETCH_LIMIT);
}
function truncateHistoryBody(text: string): string {
if (text.length <= MAX_HISTORY_BODY_CHARS) {
return text;
}
return `${text.slice(0, MAX_HISTORY_BODY_CHARS).trimEnd()}...`;
}
/**
* Fetch message history from BlueBubbles API for a specific chat.
* This provides the initial backfill for both group chats and DMs.
*/
export async function fetchBlueBubblesHistory(
chatIdentifier: string,
limit: number,
opts: BlueBubblesChatOpts = {},
): Promise<BlueBubblesHistoryFetchResult> {
const effectiveLimit = clampHistoryLimit(limit);
if (!chatIdentifier.trim() || effectiveLimit <= 0) {
return { entries: [], resolved: true };
}
let baseUrl: string;
let password: string;
try {
({ baseUrl, password } = resolveAccount(opts));
} catch {
return { entries: [], resolved: false };
}
// Try different common API patterns for fetching messages
const possiblePaths = [
`/api/v1/chat/${encodeURIComponent(chatIdentifier)}/messages?limit=${effectiveLimit}&sort=DESC`,
`/api/v1/messages?chatGuid=${encodeURIComponent(chatIdentifier)}&limit=${effectiveLimit}`,
`/api/v1/chat/${encodeURIComponent(chatIdentifier)}/message?limit=${effectiveLimit}`,
];
for (const path of possiblePaths) {
try {
const url = buildBlueBubblesApiUrl({ baseUrl, path, password });
const res = await blueBubblesFetchWithTimeout(
url,
{ method: "GET" },
opts.timeoutMs ?? 10000,
);
if (!res.ok) {
continue; // Try next path
}
const data = await res.json().catch(() => null);
if (!data) {
continue;
}
// Handle different response structures
let messages: unknown[] = [];
if (Array.isArray(data)) {
messages = data;
} else if (data.data && Array.isArray(data.data)) {
messages = data.data;
} else if (data.messages && Array.isArray(data.messages)) {
messages = data.messages;
} else {
continue;
}
const historyEntries: BlueBubblesHistoryEntry[] = [];
const maxScannedMessages = Math.min(
Math.max(effectiveLimit * HISTORY_SCAN_MULTIPLIER, effectiveLimit),
MAX_HISTORY_SCAN_MESSAGES,
);
for (let i = 0; i < messages.length && i < maxScannedMessages; i++) {
const item = messages[i];
const msg = item as BlueBubblesMessageData;
// Skip messages without text content
const text = msg.text?.trim();
if (!text) {
continue;
}
const sender = msg.is_from_me
? "me"
: msg.sender?.display_name || msg.sender?.address || msg.handle_id || "Unknown";
const timestamp = msg.date_created || msg.date_delivered;
historyEntries.push({
sender,
body: truncateHistoryBody(text),
timestamp,
messageId: msg.guid,
});
}
// Sort by timestamp (oldest first for context)
historyEntries.sort((a, b) => {
const aTime = a.timestamp || 0;
const bTime = b.timestamp || 0;
return aTime - bTime;
});
return {
entries: historyEntries.slice(0, effectiveLimit), // Ensure we don't exceed the requested limit
resolved: true,
};
} catch (error) {
// Continue to next path
continue;
}
}
// If none of the API paths worked, return empty history
return { entries: [], resolved: false };
}

View File

@@ -1,21 +1,17 @@
import type { OpenClawConfig } from "openclaw/plugin-sdk";
import {
createReplyPrefixOptions,
evictOldHistoryKeys,
logAckFailure,
logInboundDrop,
logTypingFailure,
recordPendingHistoryEntryIfEnabled,
resolveAckReaction,
resolveDmGroupAccessDecision,
resolveEffectiveAllowFromLists,
resolveControlCommandGate,
stripMarkdown,
type HistoryEntry,
} from "openclaw/plugin-sdk";
import { downloadBlueBubblesAttachment } from "./attachments.js";
import { markBlueBubblesChatRead, sendBlueBubblesTyping } from "./chat.js";
import { fetchBlueBubblesHistory } from "./history.js";
import { sendBlueBubblesMedia } from "./media-send.js";
import {
buildMessagePlaceholder,
@@ -243,178 +239,6 @@ function resolveBlueBubblesAckReaction(params: {
}
}
/**
* In-memory rolling history map keyed by account + chat identifier.
* Populated from incoming messages during the session.
* API backfill is attempted until one fetch resolves (or retries are exhausted).
*/
const chatHistories = new Map<string, HistoryEntry[]>();
type HistoryBackfillState = {
attempts: number;
firstAttemptAt: number;
nextAttemptAt: number;
resolved: boolean;
};
const historyBackfills = new Map<string, HistoryBackfillState>();
const HISTORY_BACKFILL_BASE_DELAY_MS = 5_000;
const HISTORY_BACKFILL_MAX_DELAY_MS = 2 * 60 * 1000;
const HISTORY_BACKFILL_MAX_ATTEMPTS = 6;
const HISTORY_BACKFILL_RETRY_WINDOW_MS = 30 * 60 * 1000;
const MAX_STORED_HISTORY_ENTRY_CHARS = 2_000;
const MAX_INBOUND_HISTORY_ENTRY_CHARS = 1_200;
const MAX_INBOUND_HISTORY_TOTAL_CHARS = 12_000;
function buildAccountScopedHistoryKey(accountId: string, historyIdentifier: string): string {
return `${accountId}\u0000${historyIdentifier}`;
}
function historyDedupKey(entry: HistoryEntry): string {
const messageId = entry.messageId?.trim();
if (messageId) {
return `id:${messageId}`;
}
return `fallback:${entry.sender}\u0000${entry.body}\u0000${entry.timestamp ?? ""}`;
}
function truncateHistoryBody(body: string, maxChars: number): string {
const trimmed = body.trim();
if (!trimmed) {
return "";
}
if (trimmed.length <= maxChars) {
return trimmed;
}
return `${trimmed.slice(0, maxChars).trimEnd()}...`;
}
function mergeHistoryEntries(params: {
apiEntries: HistoryEntry[];
currentEntries: HistoryEntry[];
limit: number;
}): HistoryEntry[] {
if (params.limit <= 0) {
return [];
}
const merged: HistoryEntry[] = [];
const seen = new Set<string>();
const appendUnique = (entry: HistoryEntry) => {
const key = historyDedupKey(entry);
if (seen.has(key)) {
return;
}
seen.add(key);
merged.push(entry);
};
for (const entry of params.apiEntries) {
appendUnique(entry);
}
for (const entry of params.currentEntries) {
appendUnique(entry);
}
if (merged.length <= params.limit) {
return merged;
}
return merged.slice(merged.length - params.limit);
}
function pruneHistoryBackfillState(): void {
for (const key of historyBackfills.keys()) {
if (!chatHistories.has(key)) {
historyBackfills.delete(key);
}
}
}
function markHistoryBackfillResolved(historyKey: string): void {
const state = historyBackfills.get(historyKey);
if (state) {
state.resolved = true;
historyBackfills.set(historyKey, state);
return;
}
historyBackfills.set(historyKey, {
attempts: 0,
firstAttemptAt: Date.now(),
nextAttemptAt: Number.POSITIVE_INFINITY,
resolved: true,
});
}
function planHistoryBackfillAttempt(historyKey: string, now: number): HistoryBackfillState | null {
const existing = historyBackfills.get(historyKey);
if (existing?.resolved) {
return null;
}
if (existing && now - existing.firstAttemptAt > HISTORY_BACKFILL_RETRY_WINDOW_MS) {
markHistoryBackfillResolved(historyKey);
return null;
}
if (existing && existing.attempts >= HISTORY_BACKFILL_MAX_ATTEMPTS) {
markHistoryBackfillResolved(historyKey);
return null;
}
if (existing && now < existing.nextAttemptAt) {
return null;
}
const attempts = (existing?.attempts ?? 0) + 1;
const firstAttemptAt = existing?.firstAttemptAt ?? now;
const backoffDelay = Math.min(
HISTORY_BACKFILL_BASE_DELAY_MS * 2 ** (attempts - 1),
HISTORY_BACKFILL_MAX_DELAY_MS,
);
const state: HistoryBackfillState = {
attempts,
firstAttemptAt,
nextAttemptAt: now + backoffDelay,
resolved: false,
};
historyBackfills.set(historyKey, state);
return state;
}
function buildInboundHistorySnapshot(params: {
entries: HistoryEntry[];
limit: number;
}): Array<{ sender: string; body: string; timestamp?: number }> | undefined {
if (params.limit <= 0 || params.entries.length === 0) {
return undefined;
}
const recent = params.entries.slice(-params.limit);
const selected: Array<{ sender: string; body: string; timestamp?: number }> = [];
let remainingChars = MAX_INBOUND_HISTORY_TOTAL_CHARS;
for (let i = recent.length - 1; i >= 0; i--) {
const entry = recent[i];
const body = truncateHistoryBody(entry.body, MAX_INBOUND_HISTORY_ENTRY_CHARS);
if (!body) {
continue;
}
if (selected.length > 0 && body.length > remainingChars) {
break;
}
selected.push({
sender: entry.sender,
body,
timestamp: entry.timestamp,
});
remainingChars -= body.length;
if (remainingChars <= 0) {
break;
}
}
if (selected.length === 0) {
return undefined;
}
selected.reverse();
return selected;
}
export async function processMessage(
message: NormalizedWebhookMessage,
target: WebhookTarget,
@@ -984,118 +808,9 @@ export async function processMessage(
.trim();
};
// History: in-memory rolling map with bounded API backfill retries
const historyLimit = isGroup
? (account.config.historyLimit ?? 0)
: (account.config.dmHistoryLimit ?? 0);
const historyIdentifier =
chatGuid ||
chatIdentifier ||
(chatId ? String(chatId) : null) ||
(isGroup ? null : message.senderId) ||
"";
const historyKey = historyIdentifier
? buildAccountScopedHistoryKey(account.accountId, historyIdentifier)
: "";
// Record the current message into rolling history
if (historyKey && historyLimit > 0) {
const nowMs = Date.now();
const senderLabel = message.fromMe ? "me" : message.senderName || message.senderId;
const normalizedHistoryBody = truncateHistoryBody(text, MAX_STORED_HISTORY_ENTRY_CHARS);
const currentEntries = recordPendingHistoryEntryIfEnabled({
historyMap: chatHistories,
limit: historyLimit,
historyKey,
entry: normalizedHistoryBody
? {
sender: senderLabel,
body: normalizedHistoryBody,
timestamp: message.timestamp ?? nowMs,
messageId: message.messageId ?? undefined,
}
: null,
});
pruneHistoryBackfillState();
const backfillAttempt = planHistoryBackfillAttempt(historyKey, nowMs);
if (backfillAttempt) {
try {
const backfillResult = await fetchBlueBubblesHistory(historyIdentifier, historyLimit, {
cfg: config,
accountId: account.accountId,
});
if (backfillResult.resolved) {
markHistoryBackfillResolved(historyKey);
}
if (backfillResult.entries.length > 0) {
const apiEntries: HistoryEntry[] = [];
for (const entry of backfillResult.entries) {
const body = truncateHistoryBody(entry.body, MAX_STORED_HISTORY_ENTRY_CHARS);
if (!body) {
continue;
}
apiEntries.push({
sender: entry.sender,
body,
timestamp: entry.timestamp,
messageId: entry.messageId,
});
}
const merged = mergeHistoryEntries({
apiEntries,
currentEntries:
currentEntries.length > 0 ? currentEntries : (chatHistories.get(historyKey) ?? []),
limit: historyLimit,
});
if (chatHistories.has(historyKey)) {
chatHistories.delete(historyKey);
}
chatHistories.set(historyKey, merged);
evictOldHistoryKeys(chatHistories);
logVerbose(
core,
runtime,
`backfilled ${backfillResult.entries.length} history messages for ${isGroup ? "group" : "DM"}: ${historyIdentifier}`,
);
} else if (!backfillResult.resolved) {
const remainingAttempts = HISTORY_BACKFILL_MAX_ATTEMPTS - backfillAttempt.attempts;
const nextBackoffMs = Math.max(backfillAttempt.nextAttemptAt - nowMs, 0);
logVerbose(
core,
runtime,
`history backfill unresolved for ${historyIdentifier}; retries left=${Math.max(remainingAttempts, 0)} next_in_ms=${nextBackoffMs}`,
);
}
} catch (err) {
const remainingAttempts = HISTORY_BACKFILL_MAX_ATTEMPTS - backfillAttempt.attempts;
const nextBackoffMs = Math.max(backfillAttempt.nextAttemptAt - nowMs, 0);
logVerbose(
core,
runtime,
`history backfill failed for ${historyIdentifier}: ${String(err)} (retries left=${Math.max(remainingAttempts, 0)} next_in_ms=${nextBackoffMs})`,
);
}
}
}
// Build inbound history from the in-memory map
let inboundHistory: Array<{ sender: string; body: string; timestamp?: number }> | undefined;
if (historyKey && historyLimit > 0) {
const entries = chatHistories.get(historyKey);
if (entries && entries.length > 0) {
inboundHistory = buildInboundHistorySnapshot({
entries,
limit: historyLimit,
});
}
}
const ctxPayload = core.channel.reply.finalizeInboundContext({
Body: body,
BodyForAgent: rawBody,
InboundHistory: inboundHistory,
RawBody: rawBody,
CommandBody: rawBody,
BodyForCommands: rawBody,

View File

@@ -4,7 +4,6 @@ import type { OpenClawConfig, PluginRuntime } from "openclaw/plugin-sdk";
import { removeAckReactionAfterReply, shouldAckReaction } from "openclaw/plugin-sdk";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import type { ResolvedBlueBubblesAccount } from "./accounts.js";
import { fetchBlueBubblesHistory } from "./history.js";
import {
handleBlueBubblesWebhookRequest,
registerBlueBubblesWebhookTarget,
@@ -39,10 +38,6 @@ vi.mock("./reactions.js", async () => {
};
});
vi.mock("./history.js", () => ({
fetchBlueBubblesHistory: vi.fn().mockResolvedValue({ entries: [], resolved: true }),
}));
// Mock runtime
const mockEnqueueSystemEvent = vi.fn();
const mockBuildPairingReply = vi.fn(() => "Pairing code: TESTCODE");
@@ -91,7 +86,6 @@ const mockChunkByNewline = vi.fn((text: string) => (text ? [text] : []));
const mockChunkTextWithMode = vi.fn((text: string) => (text ? [text] : []));
const mockChunkMarkdownTextWithMode = vi.fn((text: string) => (text ? [text] : []));
const mockResolveChunkMode = vi.fn(() => "length");
const mockFetchBlueBubblesHistory = vi.mocked(fetchBlueBubblesHistory);
function createMockRuntime(): PluginRuntime {
return {
@@ -361,7 +355,6 @@ describe("BlueBubbles webhook monitor", () => {
vi.clearAllMocks();
// Reset short ID state between tests for predictable behavior
_resetBlueBubblesShortIdState();
mockFetchBlueBubblesHistory.mockResolvedValue({ entries: [], resolved: true });
mockReadAllowFromStore.mockResolvedValue([]);
mockUpsertPairingRequest.mockResolvedValue({ code: "TESTCODE", created: true });
mockResolveRequireMention.mockReturnValue(false);
@@ -2998,279 +2991,6 @@ describe("BlueBubbles webhook monitor", () => {
});
});
describe("history backfill", () => {
it("scopes in-memory history by account to avoid cross-account leakage", async () => {
mockFetchBlueBubblesHistory.mockImplementation(async (_chatIdentifier, _limit, opts) => {
if (opts?.accountId === "acc-a") {
return {
resolved: true,
entries: [
{ sender: "A", body: "a-history", messageId: "a-history-1", timestamp: 1000 },
],
};
}
if (opts?.accountId === "acc-b") {
return {
resolved: true,
entries: [
{ sender: "B", body: "b-history", messageId: "b-history-1", timestamp: 1000 },
],
};
}
return { resolved: true, entries: [] };
});
const accountA: ResolvedBlueBubblesAccount = {
...createMockAccount({ dmHistoryLimit: 3, password: "password-a" }),
accountId: "acc-a",
};
const accountB: ResolvedBlueBubblesAccount = {
...createMockAccount({ dmHistoryLimit: 3, password: "password-b" }),
accountId: "acc-b",
};
const config: OpenClawConfig = {};
const core = createMockRuntime();
setBlueBubblesRuntime(core);
const unregisterA = registerBlueBubblesWebhookTarget({
account: accountA,
config,
runtime: { log: vi.fn(), error: vi.fn() },
core,
path: "/bluebubbles-webhook",
});
const unregisterB = registerBlueBubblesWebhookTarget({
account: accountB,
config,
runtime: { log: vi.fn(), error: vi.fn() },
core,
path: "/bluebubbles-webhook",
});
unregister = () => {
unregisterA();
unregisterB();
};
await handleBlueBubblesWebhookRequest(
createMockRequest("POST", "/bluebubbles-webhook?password=password-a", {
type: "new-message",
data: {
text: "message for account a",
handle: { address: "+15551234567" },
isGroup: false,
isFromMe: false,
guid: "a-msg-1",
chatGuid: "iMessage;-;+15551234567",
date: Date.now(),
},
}),
createMockResponse(),
);
await flushAsync();
await handleBlueBubblesWebhookRequest(
createMockRequest("POST", "/bluebubbles-webhook?password=password-b", {
type: "new-message",
data: {
text: "message for account b",
handle: { address: "+15551234567" },
isGroup: false,
isFromMe: false,
guid: "b-msg-1",
chatGuid: "iMessage;-;+15551234567",
date: Date.now(),
},
}),
createMockResponse(),
);
await flushAsync();
expect(mockDispatchReplyWithBufferedBlockDispatcher).toHaveBeenCalledTimes(2);
const firstCall = mockDispatchReplyWithBufferedBlockDispatcher.mock.calls[0]?.[0];
const secondCall = mockDispatchReplyWithBufferedBlockDispatcher.mock.calls[1]?.[0];
const firstHistory = (firstCall?.ctx.InboundHistory ?? []) as Array<{ body: string }>;
const secondHistory = (secondCall?.ctx.InboundHistory ?? []) as Array<{ body: string }>;
expect(firstHistory.map((entry) => entry.body)).toContain("a-history");
expect(secondHistory.map((entry) => entry.body)).toContain("b-history");
expect(secondHistory.map((entry) => entry.body)).not.toContain("a-history");
});
it("dedupes and caps merged history to dmHistoryLimit", async () => {
mockFetchBlueBubblesHistory.mockResolvedValueOnce({
resolved: true,
entries: [
{ sender: "Friend", body: "older context", messageId: "hist-1", timestamp: 1000 },
{ sender: "Friend", body: "current text", messageId: "msg-1", timestamp: 2000 },
],
});
const account = createMockAccount({ dmHistoryLimit: 2 });
const config: OpenClawConfig = {};
const core = createMockRuntime();
setBlueBubblesRuntime(core);
unregister = registerBlueBubblesWebhookTarget({
account,
config,
runtime: { log: vi.fn(), error: vi.fn() },
core,
path: "/bluebubbles-webhook",
});
const req = createMockRequest("POST", "/bluebubbles-webhook", {
type: "new-message",
data: {
text: "current text",
handle: { address: "+15551234567" },
isGroup: false,
isFromMe: false,
guid: "msg-1",
chatGuid: "iMessage;-;+15550002002",
date: Date.now(),
},
});
const res = createMockResponse();
await handleBlueBubblesWebhookRequest(req, res);
await flushAsync();
const callArgs = getFirstDispatchCall();
const inboundHistory = (callArgs.ctx.InboundHistory ?? []) as Array<{ body: string }>;
expect(inboundHistory).toHaveLength(2);
expect(inboundHistory.map((entry) => entry.body)).toEqual(["older context", "current text"]);
expect(inboundHistory.filter((entry) => entry.body === "current text")).toHaveLength(1);
});
it("uses exponential backoff for unresolved backfill and stops after resolve", async () => {
mockFetchBlueBubblesHistory
.mockResolvedValueOnce({ resolved: false, entries: [] })
.mockResolvedValueOnce({
resolved: true,
entries: [
{ sender: "Friend", body: "older context", messageId: "hist-1", timestamp: 1000 },
],
});
const account = createMockAccount({ dmHistoryLimit: 4 });
const config: OpenClawConfig = {};
const core = createMockRuntime();
setBlueBubblesRuntime(core);
unregister = registerBlueBubblesWebhookTarget({
account,
config,
runtime: { log: vi.fn(), error: vi.fn() },
core,
path: "/bluebubbles-webhook",
});
const mkPayload = (guid: string, text: string, now: number) => ({
type: "new-message",
data: {
text,
handle: { address: "+15551234567" },
isGroup: false,
isFromMe: false,
guid,
chatGuid: "iMessage;-;+15550003003",
date: now,
},
});
let now = 1_700_000_000_000;
const nowSpy = vi.spyOn(Date, "now").mockImplementation(() => now);
try {
await handleBlueBubblesWebhookRequest(
createMockRequest("POST", "/bluebubbles-webhook", mkPayload("msg-1", "first text", now)),
createMockResponse(),
);
await flushAsync();
expect(mockFetchBlueBubblesHistory).toHaveBeenCalledTimes(1);
now += 1_000;
await handleBlueBubblesWebhookRequest(
createMockRequest("POST", "/bluebubbles-webhook", mkPayload("msg-2", "second text", now)),
createMockResponse(),
);
await flushAsync();
expect(mockFetchBlueBubblesHistory).toHaveBeenCalledTimes(1);
now += 6_000;
await handleBlueBubblesWebhookRequest(
createMockRequest("POST", "/bluebubbles-webhook", mkPayload("msg-3", "third text", now)),
createMockResponse(),
);
await flushAsync();
expect(mockFetchBlueBubblesHistory).toHaveBeenCalledTimes(2);
const thirdCall = mockDispatchReplyWithBufferedBlockDispatcher.mock.calls[2]?.[0];
const thirdHistory = (thirdCall?.ctx.InboundHistory ?? []) as Array<{ body: string }>;
expect(thirdHistory.map((entry) => entry.body)).toContain("older context");
expect(thirdHistory.map((entry) => entry.body)).toContain("third text");
now += 10_000;
await handleBlueBubblesWebhookRequest(
createMockRequest("POST", "/bluebubbles-webhook", mkPayload("msg-4", "fourth text", now)),
createMockResponse(),
);
await flushAsync();
expect(mockFetchBlueBubblesHistory).toHaveBeenCalledTimes(2);
} finally {
nowSpy.mockRestore();
}
});
it("caps inbound history payload size to reduce prompt-bomb risk", async () => {
const huge = "x".repeat(8_000);
mockFetchBlueBubblesHistory.mockResolvedValueOnce({
resolved: true,
entries: Array.from({ length: 20 }, (_, idx) => ({
sender: `Friend ${idx}`,
body: `${huge} ${idx}`,
messageId: `hist-${idx}`,
timestamp: idx + 1,
})),
});
const account = createMockAccount({ dmHistoryLimit: 20 });
const config: OpenClawConfig = {};
const core = createMockRuntime();
setBlueBubblesRuntime(core);
unregister = registerBlueBubblesWebhookTarget({
account,
config,
runtime: { log: vi.fn(), error: vi.fn() },
core,
path: "/bluebubbles-webhook",
});
await handleBlueBubblesWebhookRequest(
createMockRequest("POST", "/bluebubbles-webhook", {
type: "new-message",
data: {
text: "latest text",
handle: { address: "+15551234567" },
isGroup: false,
isFromMe: false,
guid: "msg-bomb-1",
chatGuid: "iMessage;-;+15550004004",
date: Date.now(),
},
}),
createMockResponse(),
);
await flushAsync();
const callArgs = getFirstDispatchCall();
const inboundHistory = (callArgs.ctx.InboundHistory ?? []) as Array<{ body: string }>;
const totalChars = inboundHistory.reduce((sum, entry) => sum + entry.body.length, 0);
expect(inboundHistory.length).toBeLessThan(20);
expect(totalChars).toBeLessThanOrEqual(12_000);
expect(inboundHistory.every((entry) => entry.body.length <= 1_203)).toBe(true);
});
});
describe("fromMe messages", () => {
it("ignores messages from self (fromMe=true)", async () => {
const account = createMockAccount();

View File

@@ -1,58 +0,0 @@
#!/usr/bin/env bash
set -e
INSTALL_DIR="/opt/openclaw"
REPO="https://git.yalu.pro/evgeny/openclaw.git"
# ====== PORT SETTINGS ======
OPENCLAW_PORT="${OPENCLAW_PORT:-18789}"
OPENCLAW_HOST="${OPENCLAW_HOST:-0.0.0.0}"
echo "Installing OpenClaw..."
echo "Gateway will run on: ${OPENCLAW_HOST}:${OPENCLAW_PORT}"
# ====== Node 22 ======
if ! command -v node >/dev/null || [[ $(node -v | cut -d. -f1 | tr -d v) -lt 22 ]]; then
curl -fsSL https://deb.nodesource.com/setup_22.x | bash -
apt install -y nodejs
fi
# ====== pnpm ======
if ! command -v pnpm >/dev/null; then
npm install -g pnpm
fi
# ====== clone or update ======
if [ -d "$INSTALL_DIR" ]; then
git -C "$INSTALL_DIR" pull
else
git clone "$REPO" "$INSTALL_DIR"
fi
cd "$INSTALL_DIR"
pnpm install
pnpm build
echo "Running onboarding..."
npx openclaw onboard
echo "Starting OpenClaw..."
# передаём порт в runtime
export OPENCLAW_PORT
export OPENCLAW_HOST
npx openclaw start &
sleep 5
SERVER_IP=$(hostname -I | awk '{print $1}')
echo ""
echo "====================================="
echo " OpenClaw is running!"
echo ""
echo " Local: http://localhost:${OPENCLAW_PORT}"
echo " Server: http://${SERVER_IP}:${OPENCLAW_PORT}"
echo "====================================="

View File

@@ -0,0 +1,71 @@
import fs from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { describe, expect, it } from "vitest";
import {
activateSecretsRuntimeSnapshot,
clearSecretsRuntimeSnapshot,
prepareSecretsRuntimeSnapshot,
} from "../secrets/runtime.js";
import { ensureAuthProfileStore, markAuthProfileUsed } from "./auth-profiles.js";
describe("auth profile runtime snapshot persistence", () => {
it("does not write resolved plaintext keys during usage updates", async () => {
const stateDir = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-auth-runtime-save-"));
const agentDir = path.join(stateDir, "agents", "main", "agent");
const authPath = path.join(agentDir, "auth-profiles.json");
try {
await fs.mkdir(agentDir, { recursive: true });
await fs.writeFile(
authPath,
`${JSON.stringify(
{
version: 1,
profiles: {
"openai:default": {
type: "api_key",
provider: "openai",
keyRef: { source: "env", id: "OPENAI_API_KEY" },
},
},
},
null,
2,
)}\n`,
"utf8",
);
const snapshot = await prepareSecretsRuntimeSnapshot({
config: {},
env: { OPENAI_API_KEY: "sk-runtime-openai" },
agentDirs: [agentDir],
});
activateSecretsRuntimeSnapshot(snapshot);
const runtimeStore = ensureAuthProfileStore(agentDir);
expect(runtimeStore.profiles["openai:default"]).toMatchObject({
type: "api_key",
key: "sk-runtime-openai",
keyRef: { source: "env", id: "OPENAI_API_KEY" },
});
await markAuthProfileUsed({
store: runtimeStore,
profileId: "openai:default",
agentDir,
});
const persisted = JSON.parse(await fs.readFile(authPath, "utf8")) as {
profiles: Record<string, { key?: string; keyRef?: unknown }>;
};
expect(persisted.profiles["openai:default"]?.key).toBeUndefined();
expect(persisted.profiles["openai:default"]?.keyRef).toEqual({
source: "env",
id: "OPENAI_API_KEY",
});
} finally {
clearSecretsRuntimeSnapshot();
await fs.rm(stateDir, { recursive: true, force: true });
}
});
});

View File

@@ -0,0 +1,62 @@
import fs from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { describe, expect, it } from "vitest";
import { resolveAuthStorePath } from "./auth-profiles/paths.js";
import { saveAuthProfileStore } from "./auth-profiles/store.js";
import type { AuthProfileStore } from "./auth-profiles/types.js";
describe("saveAuthProfileStore", () => {
it("strips plaintext when keyRef/tokenRef are present", async () => {
const agentDir = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-auth-save-"));
try {
const store: AuthProfileStore = {
version: 1,
profiles: {
"openai:default": {
type: "api_key",
provider: "openai",
key: "sk-runtime-value",
keyRef: { source: "env", id: "OPENAI_API_KEY" },
},
"github-copilot:default": {
type: "token",
provider: "github-copilot",
token: "gh-runtime-token",
tokenRef: { source: "env", id: "GITHUB_TOKEN" },
},
"anthropic:default": {
type: "api_key",
provider: "anthropic",
key: "sk-anthropic-plain",
},
},
};
saveAuthProfileStore(store, agentDir);
const parsed = JSON.parse(await fs.readFile(resolveAuthStorePath(agentDir), "utf8")) as {
profiles: Record<
string,
{ key?: string; keyRef?: unknown; token?: string; tokenRef?: unknown }
>;
};
expect(parsed.profiles["openai:default"]?.key).toBeUndefined();
expect(parsed.profiles["openai:default"]?.keyRef).toEqual({
source: "env",
id: "OPENAI_API_KEY",
});
expect(parsed.profiles["github-copilot:default"]?.token).toBeUndefined();
expect(parsed.profiles["github-copilot:default"]?.tokenRef).toEqual({
source: "env",
id: "GITHUB_TOKEN",
});
expect(parsed.profiles["anthropic:default"]?.key).toBe("sk-anthropic-plain");
} finally {
await fs.rm(agentDir, { recursive: true, force: true });
}
});
});

View File

@@ -17,7 +17,10 @@ export {
suggestOAuthProfileIdForLegacyDefault,
} from "./auth-profiles/repair.js";
export {
clearRuntimeAuthProfileStoreSnapshots,
ensureAuthProfileStore,
loadAuthProfileStoreForRuntime,
replaceRuntimeAuthProfileStoreSnapshots,
loadAuthProfileStore,
saveAuthProfileStore,
} from "./auth-profiles/store.js";

View File

@@ -18,6 +18,65 @@ function _syncAuthProfileStore(target: AuthProfileStore, source: AuthProfileStor
target.usageStats = source.usageStats;
}
const runtimeAuthStoreSnapshots = new Map<string, AuthProfileStore>();
function resolveRuntimeStoreKey(agentDir?: string): string {
return resolveAuthStorePath(agentDir);
}
function cloneAuthProfileStore(store: AuthProfileStore): AuthProfileStore {
return structuredClone(store);
}
function resolveRuntimeAuthProfileStore(agentDir?: string): AuthProfileStore | null {
if (runtimeAuthStoreSnapshots.size === 0) {
return null;
}
const mainKey = resolveRuntimeStoreKey(undefined);
const requestedKey = resolveRuntimeStoreKey(agentDir);
const mainStore = runtimeAuthStoreSnapshots.get(mainKey);
const requestedStore = runtimeAuthStoreSnapshots.get(requestedKey);
if (!agentDir || requestedKey === mainKey) {
if (!mainStore) {
return null;
}
return cloneAuthProfileStore(mainStore);
}
if (mainStore && requestedStore) {
return mergeAuthProfileStores(
cloneAuthProfileStore(mainStore),
cloneAuthProfileStore(requestedStore),
);
}
if (requestedStore) {
return cloneAuthProfileStore(requestedStore);
}
if (mainStore) {
return cloneAuthProfileStore(mainStore);
}
return null;
}
export function replaceRuntimeAuthProfileStoreSnapshots(
entries: Array<{ agentDir?: string; store: AuthProfileStore }>,
): void {
runtimeAuthStoreSnapshots.clear();
for (const entry of entries) {
runtimeAuthStoreSnapshots.set(
resolveRuntimeStoreKey(entry.agentDir),
cloneAuthProfileStore(entry.store),
);
}
}
export function clearRuntimeAuthProfileStoreSnapshots(): void {
runtimeAuthStoreSnapshots.clear();
}
export async function updateAuthProfileStoreWithLock(params: {
agentDir?: string;
updater: (store: AuthProfileStore) => boolean;
@@ -316,10 +375,30 @@ function loadAuthProfileStoreForAgent(
return store;
}
export function loadAuthProfileStoreForRuntime(
agentDir?: string,
options?: { allowKeychainPrompt?: boolean },
): AuthProfileStore {
const store = loadAuthProfileStoreForAgent(agentDir, options);
const authPath = resolveAuthStorePath(agentDir);
const mainAuthPath = resolveAuthStorePath();
if (!agentDir || authPath === mainAuthPath) {
return store;
}
const mainStore = loadAuthProfileStoreForAgent(undefined, options);
return mergeAuthProfileStores(mainStore, store);
}
export function ensureAuthProfileStore(
agentDir?: string,
options?: { allowKeychainPrompt?: boolean },
): AuthProfileStore {
const runtimeStore = resolveRuntimeAuthProfileStore(agentDir);
if (runtimeStore) {
return runtimeStore;
}
const store = loadAuthProfileStoreForAgent(agentDir, options);
const authPath = resolveAuthStorePath(agentDir);
const mainAuthPath = resolveAuthStorePath();
@@ -335,9 +414,24 @@ export function ensureAuthProfileStore(
export function saveAuthProfileStore(store: AuthProfileStore, agentDir?: string): void {
const authPath = resolveAuthStorePath(agentDir);
const profiles = Object.fromEntries(
Object.entries(store.profiles).map(([profileId, credential]) => {
if (credential.type === "api_key" && credential.keyRef && credential.key !== undefined) {
const sanitized = { ...credential } as Record<string, unknown>;
delete sanitized.key;
return [profileId, sanitized];
}
if (credential.type === "token" && credential.tokenRef && credential.token !== undefined) {
const sanitized = { ...credential } as Record<string, unknown>;
delete sanitized.token;
return [profileId, sanitized];
}
return [profileId, credential];
}),
) as AuthProfileStore["profiles"];
const payload = {
version: AUTH_STORE_VERSION,
profiles: store.profiles,
profiles,
order: store.order ?? undefined,
lastGood: store.lastGood ?? undefined,
usageStats: store.usageStats ?? undefined,

View File

@@ -1,10 +1,12 @@
import type { OAuthCredentials } from "@mariozechner/pi-ai";
import type { OpenClawConfig } from "../../config/config.js";
import type { SecretRef } from "../../config/types.secrets.js";
export type ApiKeyCredential = {
type: "api_key";
provider: string;
key?: string;
keyRef?: SecretRef;
email?: string;
/** Optional provider-specific metadata (e.g., account IDs, gateway IDs). */
metadata?: Record<string, string>;
@@ -18,6 +20,7 @@ export type TokenCredential = {
type: "token";
provider: string;
token: string;
tokenRef?: SecretRef;
/** Optional expiry timestamp (ms since epoch). */
expires?: number;
email?: string;

View File

@@ -31,6 +31,7 @@ export function mockCatalogImportFailThenRecover() {
throw new Error("boom");
}
return {
discoverAuthStorage: () => ({}),
AuthStorage: class {},
ModelRegistry: class {
getAll() {

View File

@@ -38,6 +38,7 @@ describe("loadModelCatalog", () => {
__setModelCatalogImportForTest(
async () =>
({
discoverAuthStorage: () => ({}),
AuthStorage: class {},
ModelRegistry: class {
getAll() {
@@ -69,6 +70,7 @@ describe("loadModelCatalog", () => {
__setModelCatalogImportForTest(
async () =>
({
discoverAuthStorage: () => ({}),
AuthStorage: class {},
ModelRegistry: class {
getAll() {

View File

@@ -70,14 +70,6 @@ export function __setModelCatalogImportForTest(loader?: () => Promise<PiSdkModul
importPiSdk = loader ?? defaultImportPiSdk;
}
function createAuthStorage(AuthStorageLike: unknown, path: string) {
const withFactory = AuthStorageLike as { create?: (path: string) => unknown };
if (typeof withFactory.create === "function") {
return withFactory.create(path);
}
return new (AuthStorageLike as { new (path: string): unknown })(path);
}
export async function loadModelCatalog(params?: {
config?: OpenClawConfig;
useCache?: boolean;
@@ -102,9 +94,6 @@ export async function loadModelCatalog(params?: {
try {
const cfg = params?.config ?? loadConfig();
await ensureOpenClawModelsJson(cfg);
await (
await import("./pi-auth-json.js")
).ensurePiAuthJsonFromAuthProfiles(resolveOpenClawAgentDir());
// IMPORTANT: keep the dynamic import *inside* the try/catch.
// If this fails once (e.g. during a pnpm install that temporarily swaps node_modules),
// we must not poison the cache with a rejected promise (otherwise all channel handlers
@@ -112,7 +101,7 @@ export async function loadModelCatalog(params?: {
const piSdk = await importPiSdk();
const agentDir = resolveOpenClawAgentDir();
const { join } = await import("node:path");
const authStorage = createAuthStorage(piSdk.AuthStorage, join(agentDir, "auth.json"));
const authStorage = piSdk.discoverAuthStorage(agentDir);
const registry = new (piSdk.ModelRegistry as unknown as {
new (
authStorage: unknown,

View File

@@ -5,6 +5,7 @@ import {
DEFAULT_COPILOT_API_BASE_URL,
resolveCopilotApiToken,
} from "../providers/github-copilot-token.js";
import { normalizeOptionalSecretInput } from "../utils/normalize-secret-input.js";
import { ensureAuthProfileStore, listProfilesForProvider } from "./auth-profiles.js";
import { discoverBedrockModels } from "./bedrock-discovery.js";
import {
@@ -387,16 +388,17 @@ export function normalizeProviders(params: {
for (const [key, provider] of Object.entries(providers)) {
const normalizedKey = key.trim();
let normalizedProvider = provider;
const configuredApiKey = normalizedProvider.apiKey;
// Fix common misconfig: apiKey set to "${ENV_VAR}" instead of "ENV_VAR".
if (
normalizedProvider.apiKey &&
normalizeApiKeyConfig(normalizedProvider.apiKey) !== normalizedProvider.apiKey
typeof configuredApiKey === "string" &&
normalizeApiKeyConfig(configuredApiKey) !== configuredApiKey
) {
mutated = true;
normalizedProvider = {
...normalizedProvider,
apiKey: normalizeApiKeyConfig(normalizedProvider.apiKey),
apiKey: normalizeApiKeyConfig(configuredApiKey),
};
}
@@ -404,7 +406,9 @@ export function normalizeProviders(params: {
// Fill it from the environment or auth profiles when possible.
const hasModels =
Array.isArray(normalizedProvider.models) && normalizedProvider.models.length > 0;
if (hasModels && !normalizedProvider.apiKey?.trim()) {
const normalizedApiKey = normalizeOptionalSecretInput(normalizedProvider.apiKey);
const hasConfiguredApiKey = Boolean(normalizedApiKey || normalizedProvider.apiKey);
if (hasModels && !hasConfiguredApiKey) {
const authMode =
normalizedProvider.auth ?? (normalizedKey === "amazon-bedrock" ? "aws-sdk" : undefined);
if (authMode === "aws-sdk") {
@@ -957,17 +961,8 @@ export async function resolveImplicitCopilotProvider(params: {
}
}
// pi-coding-agent's ModelRegistry marks a model "available" only if its
// `AuthStorage` has auth configured for that provider (via auth.json/env/etc).
// Our Copilot auth lives in OpenClaw's auth-profiles store instead, so we also
// write a runtime-only auth.json entry for pi-coding-agent to pick up.
//
// This is safe because it's (1) within OpenClaw's agent dir, (2) contains the
// GitHub token (not the exchanged Copilot token), and (3) matches existing
// patterns for OAuth-like providers in pi-coding-agent.
// Note: we deliberately do not write pi-coding-agent's `auth.json` here.
// OpenClaw uses its own auth store and exchanges tokens at runtime.
// `models list` uses OpenClaw's auth heuristics for availability.
// We deliberately do not write pi-coding-agent auth.json here.
// OpenClaw keeps auth in auth-profiles and resolves runtime availability from that store.
// We intentionally do NOT define custom models for Copilot in models.json.
// pi-coding-agent treats providers with models as replacements requiring apiKey.

View File

@@ -244,40 +244,6 @@ describe("parseNdjsonStream", () => {
// Final done:true chunk has no tool_calls
expect(chunks[2].message.tool_calls).toBeUndefined();
});
it("preserves unsafe integer tool arguments as exact strings", async () => {
const reader = mockNdjsonReader([
'{"model":"m","created_at":"t","message":{"role":"assistant","content":"","tool_calls":[{"function":{"name":"send","arguments":{"target":1234567890123456789,"nested":{"thread":9223372036854775807}}}}]},"done":false}',
]);
const chunks = [];
for await (const chunk of parseNdjsonStream(reader)) {
chunks.push(chunk);
}
const args = chunks[0]?.message.tool_calls?.[0]?.function.arguments as
| { target?: unknown; nested?: { thread?: unknown } }
| undefined;
expect(args?.target).toBe("1234567890123456789");
expect(args?.nested?.thread).toBe("9223372036854775807");
});
it("keeps safe integer tool arguments as numbers", async () => {
const reader = mockNdjsonReader([
'{"model":"m","created_at":"t","message":{"role":"assistant","content":"","tool_calls":[{"function":{"name":"send","arguments":{"retries":3,"delayMs":2500}}}]},"done":false}',
]);
const chunks = [];
for await (const chunk of parseNdjsonStream(reader)) {
chunks.push(chunk);
}
const args = chunks[0]?.message.tool_calls?.[0]?.function.arguments as
| { retries?: unknown; delayMs?: unknown }
| undefined;
expect(args?.retries).toBe(3);
expect(args?.delayMs).toBe(2500);
});
});
describe("createOllamaStreamFn", () => {

View File

@@ -49,130 +49,6 @@ interface OllamaToolCall {
};
}
const MAX_SAFE_INTEGER_ABS_STR = String(Number.MAX_SAFE_INTEGER);
function isAsciiDigit(ch: string | undefined): boolean {
return ch !== undefined && ch >= "0" && ch <= "9";
}
function parseJsonNumberToken(
input: string,
start: number,
): { token: string; end: number; isInteger: boolean } | null {
let idx = start;
if (input[idx] === "-") {
idx += 1;
}
if (idx >= input.length) {
return null;
}
if (input[idx] === "0") {
idx += 1;
} else if (isAsciiDigit(input[idx]) && input[idx] !== "0") {
while (isAsciiDigit(input[idx])) {
idx += 1;
}
} else {
return null;
}
let isInteger = true;
if (input[idx] === ".") {
isInteger = false;
idx += 1;
if (!isAsciiDigit(input[idx])) {
return null;
}
while (isAsciiDigit(input[idx])) {
idx += 1;
}
}
if (input[idx] === "e" || input[idx] === "E") {
isInteger = false;
idx += 1;
if (input[idx] === "+" || input[idx] === "-") {
idx += 1;
}
if (!isAsciiDigit(input[idx])) {
return null;
}
while (isAsciiDigit(input[idx])) {
idx += 1;
}
}
return {
token: input.slice(start, idx),
end: idx,
isInteger,
};
}
function isUnsafeIntegerLiteral(token: string): boolean {
const digits = token[0] === "-" ? token.slice(1) : token;
if (digits.length < MAX_SAFE_INTEGER_ABS_STR.length) {
return false;
}
if (digits.length > MAX_SAFE_INTEGER_ABS_STR.length) {
return true;
}
return digits > MAX_SAFE_INTEGER_ABS_STR;
}
function quoteUnsafeIntegerLiterals(input: string): string {
let out = "";
let inString = false;
let escaped = false;
let idx = 0;
while (idx < input.length) {
const ch = input[idx] ?? "";
if (inString) {
out += ch;
if (escaped) {
escaped = false;
} else if (ch === "\\") {
escaped = true;
} else if (ch === '"') {
inString = false;
}
idx += 1;
continue;
}
if (ch === '"') {
inString = true;
out += ch;
idx += 1;
continue;
}
if (ch === "-" || isAsciiDigit(ch)) {
const parsed = parseJsonNumberToken(input, idx);
if (parsed) {
if (parsed.isInteger && isUnsafeIntegerLiteral(parsed.token)) {
out += `"${parsed.token}"`;
} else {
out += parsed.token;
}
idx = parsed.end;
continue;
}
}
out += ch;
idx += 1;
}
return out;
}
function parseJsonPreservingUnsafeIntegers(input: string): unknown {
return JSON.parse(quoteUnsafeIntegerLiterals(input)) as unknown;
}
// ── Ollama /api/chat response types ─────────────────────────────────────────
interface OllamaChatResponse {
@@ -386,7 +262,7 @@ export async function* parseNdjsonStream(
continue;
}
try {
yield parseJsonPreservingUnsafeIntegers(trimmed) as OllamaChatResponse;
yield JSON.parse(trimmed) as OllamaChatResponse;
} catch {
log.warn(`Skipping malformed NDJSON line: ${trimmed.slice(0, 120)}`);
}
@@ -395,7 +271,7 @@ export async function* parseNdjsonStream(
if (buffer.trim()) {
try {
yield parseJsonPreservingUnsafeIntegers(buffer.trim()) as OllamaChatResponse;
yield JSON.parse(buffer.trim()) as OllamaChatResponse;
} catch {
log.warn(`Skipping malformed trailing data: ${buffer.trim().slice(0, 120)}`);
}

View File

@@ -4,6 +4,10 @@ import { ensureAuthProfileStore } from "./auth-profiles.js";
import type { AuthProfileCredential } from "./auth-profiles/types.js";
import { normalizeProviderId } from "./model-selection.js";
/**
* @deprecated Legacy bridge for older flows that still expect `agentDir/auth.json`.
* Runtime auth resolution uses auth-profiles directly and should not depend on this module.
*/
type AuthJsonCredential =
| {
type: "api_key";
@@ -110,6 +114,8 @@ function credentialsEqual(a: AuthJsonCredential | undefined, b: AuthJsonCredenti
* registry/catalog output.
*
* Syncs all credential types: api_key, token (as api_key), and oauth.
*
* @deprecated Runtime auth now comes from OpenClaw auth-profiles snapshots.
*/
export async function ensurePiAuthJsonFromAuthProfiles(agentDir: string): Promise<{
wrote: boolean;

View File

@@ -377,11 +377,4 @@ describe("classifyFailoverReason", () => {
),
).toBe("rate_limit");
});
it("classifies JSON api_error internal server failures as timeout", () => {
expect(
classifyFailoverReason(
'{"type":"error","error":{"type":"api_error","message":"Internal server error"}}',
),
).toBe("timeout");
});
});

View File

@@ -686,16 +686,6 @@ export function isOverloadedErrorMessage(raw: string): boolean {
return matchesErrorPatterns(raw, ERROR_PATTERNS.overloaded);
}
function isJsonApiInternalServerError(raw: string): boolean {
if (!raw) {
return false;
}
const value = raw.toLowerCase();
// Anthropic often wraps transient 500s in JSON payloads like:
// {"type":"error","error":{"type":"api_error","message":"Internal server error"}}
return value.includes('"type":"api_error"') && value.includes("internal server error");
}
export function parseImageDimensionError(raw: string): {
maxDimensionPx?: number;
messageIndex?: number;
@@ -804,9 +794,6 @@ export function classifyFailoverReason(raw: string): FailoverReason | null {
// Treat transient 5xx provider failures as retryable transport issues.
return "timeout";
}
if (isJsonApiInternalServerError(raw)) {
return "timeout";
}
if (isRateLimitErrorMessage(raw)) {
return "rate_limit";
}

View File

@@ -1,76 +0,0 @@
import { describe, expect, it, vi } from "vitest";
import { createInlineCodeState } from "../markdown/code-spans.js";
import { handleAgentEnd } from "./pi-embedded-subscribe.handlers.lifecycle.js";
import type { EmbeddedPiSubscribeContext } from "./pi-embedded-subscribe.handlers.types.js";
vi.mock("../infra/agent-events.js", () => ({
emitAgentEvent: vi.fn(),
}));
function createContext(
lastAssistant: unknown,
overrides?: { onAgentEvent?: (event: unknown) => void },
): EmbeddedPiSubscribeContext {
return {
params: {
runId: "run-1",
config: {},
sessionKey: "agent:main:main",
onAgentEvent: overrides?.onAgentEvent,
},
state: {
lastAssistant: lastAssistant as EmbeddedPiSubscribeContext["state"]["lastAssistant"],
pendingCompactionRetry: 0,
blockState: {
thinking: true,
final: true,
inlineCode: createInlineCodeState(),
},
},
log: {
debug: vi.fn(),
warn: vi.fn(),
},
flushBlockReplyBuffer: vi.fn(),
resolveCompactionRetry: vi.fn(),
maybeResolveCompactionWait: vi.fn(),
} as unknown as EmbeddedPiSubscribeContext;
}
describe("handleAgentEnd", () => {
it("logs the resolved error message when run ends with assistant error", () => {
const onAgentEvent = vi.fn();
const ctx = createContext(
{
role: "assistant",
stopReason: "error",
errorMessage: "connection refused",
content: [{ type: "text", text: "" }],
},
{ onAgentEvent },
);
handleAgentEnd(ctx);
const warn = vi.mocked(ctx.log.warn);
expect(warn).toHaveBeenCalledTimes(1);
expect(warn.mock.calls[0]?.[0]).toContain("runId=run-1");
expect(warn.mock.calls[0]?.[0]).toContain("error=connection refused");
expect(onAgentEvent).toHaveBeenCalledWith({
stream: "lifecycle",
data: {
phase: "error",
error: "connection refused",
},
});
});
it("keeps non-error run-end logging on debug only", () => {
const ctx = createContext(undefined);
handleAgentEnd(ctx);
expect(ctx.log.warn).not.toHaveBeenCalled();
expect(ctx.log.debug).toHaveBeenCalledWith("embedded run agent end: runId=run-1 isError=false");
});
});

View File

@@ -29,6 +29,8 @@ export function handleAgentEnd(ctx: EmbeddedPiSubscribeContext) {
const lastAssistant = ctx.state.lastAssistant;
const isError = isAssistantMessage(lastAssistant) && lastAssistant.stopReason === "error";
ctx.log.debug(`embedded run agent end: runId=${ctx.params.runId} isError=${isError}`);
if (isError && lastAssistant) {
const friendlyError = formatAssistantErrorText(lastAssistant, {
cfg: ctx.params.config,
@@ -36,16 +38,12 @@ export function handleAgentEnd(ctx: EmbeddedPiSubscribeContext) {
provider: lastAssistant.provider,
model: lastAssistant.model,
});
const errorText = (friendlyError || lastAssistant.errorMessage || "LLM request failed.").trim();
ctx.log.warn(
`embedded run agent end: runId=${ctx.params.runId} isError=true error=${errorText}`,
);
emitAgentEvent({
runId: ctx.params.runId,
stream: "lifecycle",
data: {
phase: "error",
error: errorText,
error: friendlyError || lastAssistant.errorMessage || "LLM request failed.",
endedAt: Date.now(),
},
});
@@ -53,11 +51,10 @@ export function handleAgentEnd(ctx: EmbeddedPiSubscribeContext) {
stream: "lifecycle",
data: {
phase: "error",
error: errorText,
error: friendlyError || lastAssistant.errorMessage || "LLM request failed.",
},
});
} else {
ctx.log.debug(`embedded run agent end: runId=${ctx.params.runId} isError=${isError}`);
emitAgentEvent({
runId: ctx.params.runId,
stream: "lifecycle",

View File

@@ -0,0 +1,116 @@
import fs from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { describe, expect, it } from "vitest";
import { saveAuthProfileStore } from "./auth-profiles.js";
import { discoverAuthStorage } from "./pi-model-discovery.js";
async function createAgentDir(): Promise<string> {
return await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-pi-auth-storage-"));
}
async function pathExists(pathname: string): Promise<boolean> {
try {
await fs.stat(pathname);
return true;
} catch {
return false;
}
}
describe("discoverAuthStorage", () => {
it("loads runtime credentials from auth-profiles without writing auth.json", async () => {
const agentDir = await createAgentDir();
try {
saveAuthProfileStore(
{
version: 1,
profiles: {
"openrouter:default": {
type: "api_key",
provider: "openrouter",
key: "sk-or-v1-runtime",
},
"anthropic:default": {
type: "token",
provider: "anthropic",
token: "sk-ant-runtime",
},
"openai-codex:default": {
type: "oauth",
provider: "openai-codex",
access: "oauth-access",
refresh: "oauth-refresh",
expires: Date.now() + 60_000,
},
},
},
agentDir,
);
const authStorage = discoverAuthStorage(agentDir);
expect(authStorage.hasAuth("openrouter")).toBe(true);
expect(authStorage.hasAuth("anthropic")).toBe(true);
expect(authStorage.hasAuth("openai-codex")).toBe(true);
await expect(authStorage.getApiKey("openrouter")).resolves.toBe("sk-or-v1-runtime");
await expect(authStorage.getApiKey("anthropic")).resolves.toBe("sk-ant-runtime");
expect(authStorage.get("openai-codex")).toMatchObject({
type: "oauth",
access: "oauth-access",
});
expect(await pathExists(path.join(agentDir, "auth.json"))).toBe(false);
} finally {
await fs.rm(agentDir, { recursive: true, force: true });
}
});
it("scrubs static api_key entries from legacy auth.json and keeps oauth entries", async () => {
const agentDir = await createAgentDir();
try {
saveAuthProfileStore(
{
version: 1,
profiles: {
"openrouter:default": {
type: "api_key",
provider: "openrouter",
key: "sk-or-v1-runtime",
},
},
},
agentDir,
);
await fs.writeFile(
path.join(agentDir, "auth.json"),
JSON.stringify(
{
openrouter: { type: "api_key", key: "legacy-static-key" },
"openai-codex": {
type: "oauth",
access: "oauth-access",
refresh: "oauth-refresh",
expires: Date.now() + 60_000,
},
},
null,
2,
),
);
discoverAuthStorage(agentDir);
const parsed = JSON.parse(await fs.readFile(path.join(agentDir, "auth.json"), "utf8")) as {
[key: string]: unknown;
};
expect(parsed.openrouter).toBeUndefined();
expect(parsed["openai-codex"]).toMatchObject({
type: "oauth",
access: "oauth-access",
});
} finally {
await fs.rm(agentDir, { recursive: true, force: true });
}
});
});

View File

@@ -1,19 +1,172 @@
import fs from "node:fs";
import path from "node:path";
import { AuthStorage, ModelRegistry } from "@mariozechner/pi-coding-agent";
import {
AuthStorage,
InMemoryAuthStorageBackend,
ModelRegistry,
} from "@mariozechner/pi-coding-agent";
import { ensureAuthProfileStore } from "./auth-profiles.js";
import type { AuthProfileCredential } from "./auth-profiles.js";
import { normalizeProviderId } from "./model-selection.js";
export { AuthStorage, ModelRegistry } from "@mariozechner/pi-coding-agent";
function createAuthStorage(AuthStorageLike: unknown, path: string) {
const withFactory = AuthStorageLike as { create?: (path: string) => unknown };
if (typeof withFactory.create === "function") {
return withFactory.create(path) as AuthStorage;
type PiApiKeyCredential = { type: "api_key"; key: string };
type PiOAuthCredential = {
type: "oauth";
access: string;
refresh: string;
expires: number;
};
type PiCredential = PiApiKeyCredential | PiOAuthCredential;
type PiCredentialMap = Record<string, PiCredential>;
function isRecord(value: unknown): value is Record<string, unknown> {
return typeof value === "object" && value !== null && !Array.isArray(value);
}
function scrubLegacyStaticAuthJsonEntries(pathname: string): void {
if (!fs.existsSync(pathname)) {
return;
}
return new (AuthStorageLike as { new (path: string): unknown })(path) as AuthStorage;
let parsed: unknown;
try {
parsed = JSON.parse(fs.readFileSync(pathname, "utf8")) as unknown;
} catch {
return;
}
if (!isRecord(parsed)) {
return;
}
let changed = false;
for (const [provider, value] of Object.entries(parsed)) {
if (!isRecord(value)) {
continue;
}
if (value.type !== "api_key") {
continue;
}
delete parsed[provider];
changed = true;
}
if (!changed) {
return;
}
if (Object.keys(parsed).length === 0) {
fs.rmSync(pathname, { force: true });
return;
}
fs.writeFileSync(pathname, `${JSON.stringify(parsed, null, 2)}\n`, "utf8");
fs.chmodSync(pathname, 0o600);
}
function createAuthStorage(AuthStorageLike: unknown, path: string, creds: PiCredentialMap) {
const withInMemory = AuthStorageLike as { inMemory?: (data?: unknown) => unknown };
if (typeof withInMemory.inMemory === "function") {
return withInMemory.inMemory(creds) as AuthStorage;
}
const withFromStorage = AuthStorageLike as {
fromStorage?: (storage: unknown) => unknown;
};
if (typeof withFromStorage.fromStorage === "function") {
const backend = new InMemoryAuthStorageBackend();
backend.withLock(() => ({
result: undefined,
next: JSON.stringify(creds, null, 2),
}));
return withFromStorage.fromStorage(backend) as AuthStorage;
}
const withFactory = AuthStorageLike as { create?: (path: string) => unknown };
const withRuntimeOverride = (
typeof withFactory.create === "function"
? withFactory.create(path)
: new (AuthStorageLike as { new (path: string): unknown })(path)
) as AuthStorage & {
setRuntimeApiKey?: (provider: string, apiKey: string) => void;
};
if (typeof withRuntimeOverride.setRuntimeApiKey === "function") {
for (const [provider, credential] of Object.entries(creds)) {
if (credential.type === "api_key") {
withRuntimeOverride.setRuntimeApiKey(provider, credential.key);
continue;
}
withRuntimeOverride.setRuntimeApiKey(provider, credential.access);
}
}
return withRuntimeOverride;
}
function convertAuthProfileCredential(cred: AuthProfileCredential): PiCredential | null {
if (cred.type === "api_key") {
const key = typeof cred.key === "string" ? cred.key.trim() : "";
if (!key) {
return null;
}
return { type: "api_key", key };
}
if (cred.type === "token") {
const token = typeof cred.token === "string" ? cred.token.trim() : "";
if (!token) {
return null;
}
if (
typeof cred.expires === "number" &&
Number.isFinite(cred.expires) &&
Date.now() >= cred.expires
) {
return null;
}
return { type: "api_key", key: token };
}
if (cred.type === "oauth") {
const access = typeof cred.access === "string" ? cred.access.trim() : "";
const refresh = typeof cred.refresh === "string" ? cred.refresh.trim() : "";
if (!access || !refresh || !Number.isFinite(cred.expires) || cred.expires <= 0) {
return null;
}
return {
type: "oauth",
access,
refresh,
expires: cred.expires,
};
}
return null;
}
function resolvePiCredentials(agentDir: string): PiCredentialMap {
const store = ensureAuthProfileStore(agentDir, { allowKeychainPrompt: false });
const credentials: PiCredentialMap = {};
for (const credential of Object.values(store.profiles)) {
const provider = normalizeProviderId(String(credential.provider ?? "")).trim();
if (!provider || credentials[provider]) {
continue;
}
const converted = convertAuthProfileCredential(credential);
if (converted) {
credentials[provider] = converted;
}
}
return credentials;
}
// Compatibility helpers for pi-coding-agent 0.50+ (discover* helpers removed).
export function discoverAuthStorage(agentDir: string): AuthStorage {
return createAuthStorage(AuthStorage, path.join(agentDir, "auth.json"));
const credentials = resolvePiCredentials(agentDir);
const authPath = path.join(agentDir, "auth.json");
scrubLegacyStaticAuthJsonEntries(authPath);
return createAuthStorage(AuthStorage, authPath, credentials);
}
export function discoverModels(authStorage: AuthStorage, agentDir: string): ModelRegistry {

View File

@@ -105,9 +105,10 @@ function applySkillConfigEnvOverrides(params: {
}
}
if (normalizedPrimaryEnv && skillConfig.apiKey && !process.env[normalizedPrimaryEnv]) {
const resolvedApiKey = typeof skillConfig.apiKey === "string" ? skillConfig.apiKey.trim() : "";
if (normalizedPrimaryEnv && resolvedApiKey && !process.env[normalizedPrimaryEnv]) {
if (!pendingOverrides[normalizedPrimaryEnv]) {
pendingOverrides[normalizedPrimaryEnv] = skillConfig.apiKey;
pendingOverrides[normalizedPrimaryEnv] = resolvedApiKey;
}
}

View File

@@ -6,7 +6,6 @@ import type { OpenClawConfig } from "../config/config.js";
import { loadConfig, writeConfigFile } from "../config/config.js";
import { resolveStateDir } from "../config/paths.js";
import { resolveArchiveKind } from "../infra/archive.js";
import { enablePluginInConfig } from "../plugins/enable.js";
import { installPluginFromNpmSpec, installPluginFromPath } from "../plugins/install.js";
import { recordPluginInstall } from "../plugins/installs.js";
import { clearPluginManifestRegistryCache } from "../plugins/manifest-registry.js";
@@ -136,6 +135,22 @@ function createPluginInstallLogger(): { info: (msg: string) => void; warn: (msg:
};
}
function enablePluginInConfig(config: OpenClawConfig, pluginId: string): OpenClawConfig {
return {
...config,
plugins: {
...config.plugins,
entries: {
...config.plugins?.entries,
[pluginId]: {
...(config.plugins?.entries?.[pluginId] as object | undefined),
enabled: true,
},
},
},
};
}
function logSlotWarnings(warnings: string[]) {
if (warnings.length === 0) {
return;
@@ -337,21 +352,24 @@ export function registerPluginsCli(program: Command) {
.argument("<id>", "Plugin id")
.action(async (id: string) => {
const cfg = loadConfig();
const enableResult = enablePluginInConfig(cfg, id);
let next: OpenClawConfig = enableResult.config;
let next: OpenClawConfig = {
...cfg,
plugins: {
...cfg.plugins,
entries: {
...cfg.plugins?.entries,
[id]: {
...(cfg.plugins?.entries as Record<string, { enabled?: boolean }> | undefined)?.[id],
enabled: true,
},
},
},
};
const slotResult = applySlotSelectionForPlugin(next, id);
next = slotResult.config;
await writeConfigFile(next);
logSlotWarnings(slotResult.warnings);
if (enableResult.enabled) {
defaultRuntime.log(`Enabled plugin "${id}". Restart the gateway to apply.`);
return;
}
defaultRuntime.log(
theme.warn(
`Plugin "${id}" could not be enabled (${enableResult.reason ?? "unknown reason"}).`,
),
);
defaultRuntime.log(`Enabled plugin "${id}". Restart the gateway to apply.`);
});
plugins
@@ -550,7 +568,7 @@ export function registerPluginsCli(program: Command) {
},
},
probe.pluginId,
).config;
);
next = recordPluginInstall(next, {
pluginId: probe.pluginId,
source: "path",
@@ -579,7 +597,7 @@ export function registerPluginsCli(program: Command) {
// force a rescan so config validation sees the freshly installed plugin.
clearPluginManifestRegistryCache();
let next = enablePluginInConfig(cfg, result.pluginId).config;
let next = enablePluginInConfig(cfg, result.pluginId);
const source: "archive" | "path" = resolveArchiveKind(resolved) ? "archive" : "path";
next = recordPluginInstall(next, {
pluginId: result.pluginId,
@@ -630,7 +648,7 @@ export function registerPluginsCli(program: Command) {
// Ensure config validation sees newly installed plugin(s) even if the cache was warmed at startup.
clearPluginManifestRegistryCache();
let next = enablePluginInConfig(cfg, result.pluginId).config;
let next = enablePluginInConfig(cfg, result.pluginId);
const resolvedSpec = result.npmResolution?.resolvedSpec;
const recordSpec = opts.pin && resolvedSpec ? resolvedSpec : raw;
if (opts.pin && !resolvedSpec) {

View File

@@ -260,6 +260,15 @@ const entries: SubCliEntry[] = [
mod.registerSecurityCli(program);
},
},
{
name: "secrets",
description: "Secrets runtime reload controls",
hasSubcommands: true,
register: async (program) => {
const mod = await import("../secrets-cli.js");
mod.registerSecretsCli(program);
},
},
{
name: "skills",
description: "List and inspect available skills",

View File

@@ -0,0 +1,96 @@
import { Command } from "commander";
import { beforeEach, describe, expect, it, vi } from "vitest";
import { createCliRuntimeCapture } from "./test-runtime-capture.js";
const callGatewayFromCli = vi.fn();
const runSecretsMigration = vi.fn();
const rollbackSecretsMigration = vi.fn();
const { defaultRuntime, runtimeLogs, runtimeErrors, resetRuntimeCapture } =
createCliRuntimeCapture();
vi.mock("./gateway-rpc.js", () => ({
addGatewayClientOptions: (cmd: Command) => cmd,
callGatewayFromCli: (method: string, opts: unknown, params?: unknown, extra?: unknown) =>
callGatewayFromCli(method, opts, params, extra),
}));
vi.mock("../runtime.js", () => ({
defaultRuntime,
}));
vi.mock("../secrets/migrate.js", () => ({
runSecretsMigration: (options: unknown) => runSecretsMigration(options),
rollbackSecretsMigration: (options: unknown) => rollbackSecretsMigration(options),
}));
const { registerSecretsCli } = await import("./secrets-cli.js");
describe("secrets CLI", () => {
const createProgram = () => {
const program = new Command();
program.exitOverride();
registerSecretsCli(program);
return program;
};
beforeEach(() => {
resetRuntimeCapture();
callGatewayFromCli.mockReset();
runSecretsMigration.mockReset();
rollbackSecretsMigration.mockReset();
});
it("calls secrets.reload and prints human output", async () => {
callGatewayFromCli.mockResolvedValue({ ok: true, warningCount: 1 });
await createProgram().parseAsync(["secrets", "reload"], { from: "user" });
expect(callGatewayFromCli).toHaveBeenCalledWith(
"secrets.reload",
expect.anything(),
undefined,
expect.objectContaining({ expectFinal: false }),
);
expect(runtimeLogs.at(-1)).toBe("Secrets reloaded with 1 warning(s).");
expect(runtimeErrors).toHaveLength(0);
});
it("prints JSON when requested", async () => {
callGatewayFromCli.mockResolvedValue({ ok: true, warningCount: 0 });
await createProgram().parseAsync(["secrets", "reload", "--json"], { from: "user" });
expect(runtimeLogs.at(-1)).toContain('"ok": true');
});
it("runs secrets migrate as dry-run by default", async () => {
runSecretsMigration.mockResolvedValue({
mode: "dry-run",
changed: true,
secretsFilePath: "/tmp/secrets.enc.json",
counters: { secretsWritten: 3 },
changedFiles: ["/tmp/openclaw.json"],
});
await createProgram().parseAsync(["secrets", "migrate"], { from: "user" });
expect(runSecretsMigration).toHaveBeenCalledWith(
expect.objectContaining({ write: false, scrubEnv: true }),
);
expect(runtimeLogs.at(-1)).toContain("dry run");
});
it("runs rollback when --rollback is provided", async () => {
rollbackSecretsMigration.mockResolvedValue({
backupId: "20260221T010203Z",
restoredFiles: 2,
deletedFiles: 1,
});
await createProgram().parseAsync(["secrets", "migrate", "--rollback", "20260221T010203Z"], {
from: "user",
});
expect(rollbackSecretsMigration).toHaveBeenCalledWith({
backupId: "20260221T010203Z",
});
expect(runtimeLogs.at(-1)).toContain("rollback complete");
});
});

121
src/cli/secrets-cli.ts Normal file
View File

@@ -0,0 +1,121 @@
import type { Command } from "commander";
import { danger } from "../globals.js";
import { defaultRuntime } from "../runtime.js";
import {
rollbackSecretsMigration,
runSecretsMigration,
type SecretsMigrationRollbackResult,
type SecretsMigrationRunResult,
} from "../secrets/migrate.js";
import { formatDocsLink } from "../terminal/links.js";
import { theme } from "../terminal/theme.js";
import { addGatewayClientOptions, callGatewayFromCli, type GatewayRpcOpts } from "./gateway-rpc.js";
type SecretsReloadOptions = GatewayRpcOpts & { json?: boolean };
type SecretsMigrateOptions = {
write?: boolean;
rollback?: string;
scrubEnv?: boolean;
json?: boolean;
};
function printMigrationResult(
result: SecretsMigrationRunResult | SecretsMigrationRollbackResult,
json: boolean,
): void {
if (json) {
defaultRuntime.log(JSON.stringify(result, null, 2));
return;
}
if ("restoredFiles" in result) {
defaultRuntime.log(
`Secrets rollback complete for backup ${result.backupId}. Restored ${result.restoredFiles} file(s), deleted ${result.deletedFiles} file(s).`,
);
return;
}
if (result.mode === "dry-run") {
if (!result.changed) {
defaultRuntime.log("Secrets migrate dry run: no changes needed.");
return;
}
defaultRuntime.log(
`Secrets migrate dry run: ${result.changedFiles.length} file(s) would change, ${result.counters.secretsWritten} secret value(s) would move to ${result.secretsFilePath}.`,
);
return;
}
if (!result.changed) {
defaultRuntime.log("Secrets migrate: no changes applied.");
return;
}
defaultRuntime.log(
`Secrets migrated. Backup: ${result.backupId}. Moved ${result.counters.secretsWritten} secret value(s) into ${result.secretsFilePath}.`,
);
}
export function registerSecretsCli(program: Command) {
const secrets = program
.command("secrets")
.description("Secrets runtime controls")
.addHelpText(
"after",
() =>
`\n${theme.muted("Docs:")} ${formatDocsLink("/gateway/security", "docs.openclaw.ai/gateway/security")}\n`,
);
addGatewayClientOptions(
secrets
.command("reload")
.description("Re-resolve secret references and atomically swap runtime snapshot")
.option("--json", "Output JSON", false),
).action(async (opts: SecretsReloadOptions) => {
try {
const result = await callGatewayFromCli("secrets.reload", opts, undefined, {
expectFinal: false,
});
if (opts.json) {
defaultRuntime.log(JSON.stringify(result, null, 2));
return;
}
const warningCount = Number(
(result as { warningCount?: unknown } | undefined)?.warningCount ?? 0,
);
if (Number.isFinite(warningCount) && warningCount > 0) {
defaultRuntime.log(`Secrets reloaded with ${warningCount} warning(s).`);
return;
}
defaultRuntime.log("Secrets reloaded.");
} catch (err) {
defaultRuntime.error(danger(String(err)));
defaultRuntime.exit(1);
}
});
secrets
.command("migrate")
.description("Migrate plaintext secrets to file-backed SecretRefs (sops)")
.option("--write", "Apply migration changes (default is dry-run)", false)
.option("--rollback <backup-id>", "Rollback a previous migration backup id")
.option("--no-scrub-env", "Keep matching plaintext values in ~/.openclaw/.env")
.option("--json", "Output JSON", false)
.action(async (opts: SecretsMigrateOptions) => {
try {
if (typeof opts.rollback === "string" && opts.rollback.trim()) {
const result = await rollbackSecretsMigration({ backupId: opts.rollback.trim() });
printMigrationResult(result, Boolean(opts.json));
return;
}
const result = await runSecretsMigration({
write: Boolean(opts.write),
scrubEnv: opts.scrubEnv ?? true,
});
printMigrationResult(result, Boolean(opts.json));
} catch (err) {
defaultRuntime.error(danger(String(err)));
defaultRuntime.exit(1);
}
});
}

View File

@@ -95,7 +95,7 @@ describe("models list auth-profile sync", () => {
const openrouter = getProviderRow(String(runtime.log.mock.calls[0]?.[0]), "openrouter/");
expect(openrouter).toBeDefined();
expect(openrouter?.available).toBe(true);
expect(await pathExists(authPath)).toBe(true);
expect(await pathExists(authPath)).toBe(false);
});
});

View File

@@ -6,9 +6,6 @@ let toModelRow: typeof import("./models/list.registry.js").toModelRow;
const loadConfig = vi.fn();
const ensureOpenClawModelsJson = vi.fn().mockResolvedValue(undefined);
const ensurePiAuthJsonFromAuthProfiles = vi
.fn()
.mockResolvedValue({ wrote: false, authPath: "/tmp/openclaw-agent/auth.json" });
const resolveOpenClawAgentDir = vi.fn().mockReturnValue("/tmp/openclaw-agent");
const ensureAuthProfileStore = vi.fn().mockReturnValue({ version: 1, profiles: {} });
const listProfilesForProvider = vi.fn().mockReturnValue([]);
@@ -38,10 +35,6 @@ vi.mock("../agents/models-config.js", () => ({
ensureOpenClawModelsJson,
}));
vi.mock("../agents/pi-auth-json.js", () => ({
ensurePiAuthJsonFromAuthProfiles,
}));
vi.mock("../agents/agent-paths.js", () => ({
resolveOpenClawAgentDir,
}));
@@ -110,7 +103,6 @@ beforeEach(() => {
modelRegistryState.getAllError = undefined;
modelRegistryState.getAvailableError = undefined;
listProfilesForProvider.mockReturnValue([]);
ensurePiAuthJsonFromAuthProfiles.mockClear();
});
afterEach(() => {
@@ -279,13 +271,12 @@ describe("models list/status", () => {
({ loadModelRegistry, toModelRow } = await import("./models/list.registry.js"));
});
it("models list syncs auth-profiles into auth.json before availability checks", async () => {
it("models list runs model discovery without auth.json sync", async () => {
setDefaultZaiRegistry();
const runtime = makeRuntime();
await modelsListCommand({ all: true, json: true }, runtime);
expect(ensurePiAuthJsonFromAuthProfiles).toHaveBeenCalledWith("/tmp/openclaw-agent");
expect(runtime.error).not.toHaveBeenCalled();
});
it("models list outputs canonical zai key for configured z.ai model", async () => {

View File

@@ -12,7 +12,6 @@ import {
resolveForwardCompatModel,
} from "../../agents/model-forward-compat.js";
import { ensureOpenClawModelsJson } from "../../agents/models-config.js";
import { ensurePiAuthJsonFromAuthProfiles } from "../../agents/pi-auth-json.js";
import type { ModelRegistry } from "../../agents/pi-model-discovery.js";
import { discoverAuthStorage, discoverModels } from "../../agents/pi-model-discovery.js";
import type { OpenClawConfig } from "../../config/config.js";
@@ -102,7 +101,6 @@ function loadAvailableModels(registry: ModelRegistry): Model<Api>[] {
export async function loadModelRegistry(cfg: OpenClawConfig) {
await ensureOpenClawModelsJson(cfg);
const agentDir = resolveOpenClawAgentDir();
await ensurePiAuthJsonFromAuthProfiles(agentDir);
const authStorage = discoverAuthStorage(agentDir);
const registry = discoverModels(authStorage, agentDir);
const appended = appendAntigravityForwardCompatModels(registry.getAll(), registry);

View File

@@ -4,6 +4,7 @@ import type { OpenClawConfig } from "../config/config.js";
import type { ModelProviderConfig } from "../config/types.models.js";
import type { RuntimeEnv } from "../runtime.js";
import { fetchWithTimeout } from "../utils/fetch-timeout.js";
import { normalizeOptionalSecretInput } from "../utils/normalize-secret-input.js";
import type { WizardPrompter } from "../wizard/prompts.js";
import { applyPrimaryModel } from "./model-picker.js";
import { normalizeAlias } from "./models/shared.js";
@@ -521,7 +522,7 @@ export function applyCustomApiConfig(params: ApplyCustomApiConfigParams): Custom
const mergedModels = hasModel ? existingModels : [...existingModels, nextModel];
const { apiKey: existingApiKey, ...existingProviderRest } = existingProvider ?? {};
const normalizedApiKey =
params.apiKey?.trim() || (existingApiKey ? existingApiKey.trim() : undefined);
normalizeOptionalSecretInput(params.apiKey) ?? normalizeOptionalSecretInput(existingApiKey);
let config: OpenClawConfig = {
...params.config,

View File

@@ -0,0 +1,74 @@
import { describe, expect, it } from "vitest";
import { validateConfigObjectRaw } from "./validation.js";
describe("config secret refs schema", () => {
it("accepts top-level secrets sources and model apiKey refs", () => {
const result = validateConfigObjectRaw({
secrets: {
sources: {
env: { type: "env" },
file: { type: "sops", path: "~/.openclaw/secrets.enc.json", timeoutMs: 10_000 },
},
},
models: {
providers: {
openai: {
baseUrl: "https://api.openai.com/v1",
apiKey: { source: "env", id: "OPENAI_API_KEY" },
models: [{ id: "gpt-5", name: "gpt-5" }],
},
},
},
});
expect(result.ok).toBe(true);
});
it("accepts googlechat serviceAccount refs", () => {
const result = validateConfigObjectRaw({
channels: {
googlechat: {
serviceAccountRef: { source: "file", id: "/channels/googlechat/serviceAccount" },
},
},
});
expect(result.ok).toBe(true);
});
it("accepts skills entry apiKey refs", () => {
const result = validateConfigObjectRaw({
skills: {
entries: {
"review-pr": {
enabled: true,
apiKey: { source: "env", id: "SKILL_REVIEW_PR_API_KEY" },
},
},
},
});
expect(result.ok).toBe(true);
});
it("rejects invalid secret ref id", () => {
const result = validateConfigObjectRaw({
models: {
providers: {
openai: {
baseUrl: "https://api.openai.com/v1",
apiKey: { source: "env", id: "bad id with spaces" },
models: [{ id: "gpt-5", name: "gpt-5" }],
},
},
},
});
expect(result.ok).toBe(false);
if (!result.ok) {
expect(
result.issues.some((issue) => issue.path.includes("models.providers.openai.apiKey")),
).toBe(true);
}
});
});

View File

@@ -1,11 +1,14 @@
export {
clearConfigCache,
clearRuntimeConfigSnapshot,
createConfigIO,
getRuntimeConfigSnapshot,
loadConfig,
parseConfigJson5,
readConfigFileSnapshot,
readConfigFileSnapshotForWrite,
resolveConfigSnapshotHash,
setRuntimeConfigSnapshot,
writeConfigFile,
} from "./io.js";
export { migrateLegacyConfig } from "./legacy-migrate.js";

View File

@@ -0,0 +1,63 @@
import fs from "node:fs/promises";
import path from "node:path";
import { describe, expect, it } from "vitest";
import { withTempHome } from "./home-env.test-harness.js";
import {
clearConfigCache,
clearRuntimeConfigSnapshot,
loadConfig,
setRuntimeConfigSnapshot,
writeConfigFile,
} from "./io.js";
import type { OpenClawConfig } from "./types.js";
describe("runtime config snapshot writes", () => {
it("preserves source secret refs when writeConfigFile receives runtime-resolved config", async () => {
await withTempHome("openclaw-config-runtime-write-", async (home) => {
const configPath = path.join(home, ".openclaw", "openclaw.json");
const sourceConfig: OpenClawConfig = {
models: {
providers: {
openai: {
baseUrl: "https://api.openai.com/v1",
apiKey: { source: "env", id: "OPENAI_API_KEY" },
models: [],
},
},
},
};
const runtimeConfig: OpenClawConfig = {
models: {
providers: {
openai: {
baseUrl: "https://api.openai.com/v1",
apiKey: "sk-runtime-resolved",
models: [],
},
},
},
};
await fs.mkdir(path.dirname(configPath), { recursive: true });
await fs.writeFile(configPath, `${JSON.stringify(sourceConfig, null, 2)}\n`, "utf8");
try {
setRuntimeConfigSnapshot(runtimeConfig, sourceConfig);
expect(loadConfig().models?.providers?.openai?.apiKey).toBe("sk-runtime-resolved");
await writeConfigFile(loadConfig());
const persisted = JSON.parse(await fs.readFile(configPath, "utf8")) as {
models?: { providers?: { openai?: { apiKey?: unknown } } };
};
expect(persisted.models?.providers?.openai?.apiKey).toEqual({
source: "env",
id: "OPENAI_API_KEY",
});
} finally {
clearRuntimeConfigSnapshot();
clearConfigCache();
}
});
});
});

View File

@@ -1061,6 +1061,8 @@ let configCache: {
expiresAt: number;
config: OpenClawConfig;
} | null = null;
let runtimeConfigSnapshot: OpenClawConfig | null = null;
let runtimeConfigSourceSnapshot: OpenClawConfig | null = null;
function resolveConfigCacheMs(env: NodeJS.ProcessEnv): number {
const raw = env.OPENCLAW_CONFIG_CACHE_MS?.trim();
@@ -1088,7 +1090,29 @@ export function clearConfigCache(): void {
configCache = null;
}
export function setRuntimeConfigSnapshot(
config: OpenClawConfig,
sourceConfig?: OpenClawConfig,
): void {
runtimeConfigSnapshot = config;
runtimeConfigSourceSnapshot = sourceConfig ?? null;
clearConfigCache();
}
export function clearRuntimeConfigSnapshot(): void {
runtimeConfigSnapshot = null;
runtimeConfigSourceSnapshot = null;
clearConfigCache();
}
export function getRuntimeConfigSnapshot(): OpenClawConfig | null {
return runtimeConfigSnapshot;
}
export function loadConfig(): OpenClawConfig {
if (runtimeConfigSnapshot) {
return runtimeConfigSnapshot;
}
const io = createConfigIO();
const configPath = io.configPath;
const now = Date.now();
@@ -1125,9 +1149,14 @@ export async function writeConfigFile(
options: ConfigWriteOptions = {},
): Promise<void> {
const io = createConfigIO();
let nextCfg = cfg;
if (runtimeConfigSnapshot && runtimeConfigSourceSnapshot) {
const runtimePatch = createMergePatch(runtimeConfigSnapshot, cfg);
nextCfg = coerceConfig(applyMergePatch(runtimeConfigSourceSnapshot, runtimePatch));
}
const sameConfigPath =
options.expectedConfigPath === undefined || options.expectedConfigPath === io.configPath;
await io.writeConfigFile(cfg, {
await io.writeConfigFile(nextCfg, {
envSnapshotForRestore: sameConfigPath ? options.envSnapshotForRestore : undefined,
});
}

View File

@@ -74,7 +74,6 @@ describe("redactConfigSnapshot", () => {
},
shortSecret: { token: "short" },
});
const result = redactConfigSnapshot(snapshot);
const cfg = result.config as typeof snapshot.config;
@@ -91,6 +90,45 @@ describe("redactConfigSnapshot", () => {
expect(cfg.shortSecret.token).toBe(REDACTED_SENTINEL);
});
it("redacts googlechat serviceAccount object payloads", () => {
const snapshot = makeSnapshot({
channels: {
googlechat: {
serviceAccount: {
type: "service_account",
client_email: "bot@example.iam.gserviceaccount.com",
private_key: "-----BEGIN PRIVATE KEY-----secret-----END PRIVATE KEY-----",
},
},
},
});
const result = redactConfigSnapshot(snapshot);
const channels = result.config.channels as Record<string, Record<string, unknown>>;
expect(channels.googlechat.serviceAccount).toBe(REDACTED_SENTINEL);
});
it("redacts object-valued apiKey refs in model providers", () => {
const snapshot = makeSnapshot({
models: {
providers: {
openai: {
apiKey: { source: "env", id: "OPENAI_API_KEY" },
baseUrl: "https://api.openai.com",
},
},
},
});
const result = redactConfigSnapshot(snapshot);
const models = result.config.models as Record<string, Record<string, Record<string, unknown>>>;
expect(models.providers.openai.apiKey).toEqual({
source: REDACTED_SENTINEL,
id: REDACTED_SENTINEL,
});
expect(models.providers.openai.baseUrl).toBe("https://api.openai.com");
});
it("preserves non-sensitive fields", () => {
const snapshot = makeSnapshot({
ui: { seamColor: "#0088cc" },

View File

@@ -17,6 +17,31 @@ function isEnvVarPlaceholder(value: string): boolean {
return ENV_VAR_PLACEHOLDER_PATTERN.test(value.trim());
}
function isWholeObjectSensitivePath(path: string): boolean {
const lowered = path.toLowerCase();
return lowered.endsWith("serviceaccount") || lowered.endsWith("serviceaccountref");
}
function collectSensitiveStrings(value: unknown, values: string[]): void {
if (typeof value === "string") {
if (!isEnvVarPlaceholder(value)) {
values.push(value);
}
return;
}
if (Array.isArray(value)) {
for (const item of value) {
collectSensitiveStrings(item, values);
}
return;
}
if (value && typeof value === "object") {
for (const item of Object.values(value as Record<string, unknown>)) {
collectSensitiveStrings(item, values);
}
}
}
function isExtensionPath(path: string): boolean {
return (
path === "plugins" ||
@@ -159,7 +184,19 @@ function redactObjectWithLookup(
result[key] = REDACTED_SENTINEL;
values.push(value);
} else if (typeof value === "object" && value !== null) {
result[key] = redactObjectWithLookup(value, lookup, candidate, values, hints);
if (hints[candidate]?.sensitive === true && !Array.isArray(value)) {
collectSensitiveStrings(value, values);
result[key] = REDACTED_SENTINEL;
} else {
result[key] = redactObjectWithLookup(value, lookup, candidate, values, hints);
}
} else if (
hints[candidate]?.sensitive === true &&
value !== undefined &&
value !== null
) {
// Keep primitives at explicitly-sensitive paths fully redacted.
result[key] = REDACTED_SENTINEL;
}
break;
}
@@ -228,6 +265,16 @@ function redactObjectGuessing(
) {
result[key] = REDACTED_SENTINEL;
values.push(value);
} else if (
!isExplicitlyNonSensitivePath(hints, [dotPath, wildcardPath]) &&
isSensitivePath(dotPath) &&
isWholeObjectSensitivePath(dotPath) &&
value &&
typeof value === "object" &&
!Array.isArray(value)
) {
collectSensitiveStrings(value, values);
result[key] = REDACTED_SENTINEL;
} else if (typeof value === "object" && value !== null) {
result[key] = redactObjectGuessing(value, dotPath, values, hints);
} else {

View File

@@ -236,13 +236,6 @@ export const FIELD_HELP: Record<string, string> = {
"memory.backend": 'Memory backend ("builtin" for OpenClaw embeddings, "qmd" for QMD sidecar).',
"memory.citations": 'Default citation behavior ("auto", "on", or "off").',
"memory.qmd.command": "Path to the qmd binary (default: resolves from PATH).",
"memory.qmd.mcporter":
"Optional: route QMD searches through mcporter (MCP runtime) instead of spawning `qmd` per query. Intended to avoid per-search cold starts when QMD models are large.",
"memory.qmd.mcporter.enabled": "Enable mcporter-backed QMD searches (default: false).",
"memory.qmd.mcporter.serverName":
"mcporter server name to call (default: qmd). Server should run `qmd mcp` with lifecycle keep-alive.",
"memory.qmd.mcporter.startDaemon":
"Start `mcporter daemon start` automatically when enabled (default: true).",
"memory.qmd.includeDefaultMemory":
"Whether to automatically index MEMORY.md + memory/**/*.md (default: true).",
"memory.qmd.paths":

View File

@@ -109,6 +109,7 @@ describe("mapSensitivePaths", () => {
expect(hints["agents.defaults.memorySearch.remote.apiKey"]?.sensitive).toBe(true);
expect(hints["agents.list[].memorySearch.remote.apiKey"]?.sensitive).toBe(true);
expect(hints["channels.discord.accounts.*.token"]?.sensitive).toBe(true);
expect(hints["channels.googlechat.serviceAccount"]?.sensitive).toBe(true);
expect(hints["gateway.auth.token"]?.sensitive).toBe(true);
expect(hints["skills.entries.*.apiKey"]?.sensitive).toBe(true);
});

View File

@@ -107,7 +107,13 @@ const NORMALIZED_SENSITIVE_KEY_WHITELIST_SUFFIXES = SENSITIVE_KEY_WHITELIST_SUFF
suffix.toLowerCase(),
);
const SENSITIVE_PATTERNS = [/token$/i, /password/i, /secret/i, /api.?key/i];
const SENSITIVE_PATTERNS = [
/token$/i,
/password/i,
/secret/i,
/api.?key/i,
/serviceaccount(?:ref)?$/i,
];
function isWhitelistedSensitivePath(path: string): boolean {
const lowerPath = path.toLowerCase();

View File

@@ -5,6 +5,7 @@ import type {
ReplyToMode,
} from "./types.base.js";
import type { DmConfig } from "./types.messages.js";
import type { SecretRef } from "./types.secrets.js";
export type GoogleChatDmConfig = {
/** If false, ignore all incoming Google Chat DMs. Default: true. */
@@ -58,8 +59,10 @@ export type GoogleChatAccountConfig = {
defaultTo?: string;
/** Per-space configuration keyed by space id or name. */
groups?: Record<string, GoogleChatGroupConfig>;
/** Service account JSON (inline string or object). */
serviceAccount?: string | Record<string, unknown>;
/** Service account JSON (inline string, object, or secret reference). */
serviceAccount?: string | Record<string, unknown> | SecretRef;
/** Explicit secret reference for service account JSON. */
serviceAccountRef?: SecretRef;
/** Service account JSON file path. */
serviceAccountFile?: string;
/** Webhook audience type (app-url or project-number). */

View File

@@ -12,7 +12,6 @@ export type MemoryConfig = {
export type MemoryQmdConfig = {
command?: string;
mcporter?: MemoryQmdMcporterConfig;
searchMode?: MemoryQmdSearchMode;
includeDefaultMemory?: boolean;
paths?: MemoryQmdIndexPath[];
@@ -22,20 +21,6 @@ export type MemoryQmdConfig = {
scope?: SessionSendPolicyConfig;
};
export type MemoryQmdMcporterConfig = {
/**
* Route QMD searches through mcporter (MCP runtime) instead of spawning `qmd` per query.
* Requires:
* - `mcporter` installed and on PATH
* - A configured mcporter server that runs `qmd mcp` with `lifecycle: keep-alive`
*/
enabled?: boolean;
/** mcporter server name (defaults to "qmd") */
serverName?: string;
/** Start the mcporter daemon automatically (defaults to true when enabled). */
startDaemon?: boolean;
};
export type MemoryQmdIndexPath = {
path: string;
name?: string;

View File

@@ -1,3 +1,5 @@
import type { SecretInput } from "./types.secrets.js";
export type ModelApi =
| "openai-completions"
| "openai-responses"
@@ -43,7 +45,7 @@ export type ModelDefinitionConfig = {
export type ModelProviderConfig = {
baseUrl: string;
apiKey?: string;
apiKey?: SecretInput;
auth?: ModelProviderAuthMode;
api?: ModelApi;
headers?: Record<string, string>;

View File

@@ -22,6 +22,7 @@ import type {
import type { ModelsConfig } from "./types.models.js";
import type { NodeHostConfig } from "./types.node-host.js";
import type { PluginsConfig } from "./types.plugins.js";
import type { SecretsConfig } from "./types.secrets.js";
import type { SkillsConfig } from "./types.skills.js";
import type { ToolsConfig } from "./types.tools.js";
@@ -75,6 +76,7 @@ export type OpenClawConfig = {
avatar?: string;
};
};
secrets?: SecretsConfig;
skills?: SkillsConfig;
plugins?: PluginsConfig;
models?: ModelsConfig;

View File

@@ -0,0 +1,49 @@
export type SecretRefSource = "env" | "file";
/**
* Stable identifier for a secret in a configured source.
* Examples:
* - env source: "OPENAI_API_KEY"
* - file source: "/providers/openai/api_key" (JSON pointer)
*/
export type SecretRef = {
source: SecretRefSource;
id: string;
};
export type SecretInput = string | SecretRef;
function isRecord(value: unknown): value is Record<string, unknown> {
return typeof value === "object" && value !== null && !Array.isArray(value);
}
export function isSecretRef(value: unknown): value is SecretRef {
if (!isRecord(value)) {
return false;
}
if (Object.keys(value).length !== 2) {
return false;
}
return (
(value.source === "env" || value.source === "file") &&
typeof value.id === "string" &&
value.id.trim().length > 0
);
}
export type EnvSecretSourceConfig = {
type?: "env";
};
export type SopsSecretSourceConfig = {
type: "sops";
path: string;
timeoutMs?: number;
};
export type SecretsConfig = {
sources?: {
env?: EnvSecretSourceConfig;
file?: SopsSecretSourceConfig;
};
};

View File

@@ -1,6 +1,8 @@
import type { SecretInput } from "./types.secrets.js";
export type SkillConfig = {
enabled?: boolean;
apiKey?: string;
apiKey?: SecretInput;
env?: Record<string, string>;
config?: Record<string, unknown>;
};

View File

@@ -22,6 +22,7 @@ export * from "./types.msteams.js";
export * from "./types.plugins.js";
export * from "./types.queue.js";
export * from "./types.sandbox.js";
export * from "./types.secrets.js";
export * from "./types.signal.js";
export * from "./types.skills.js";
export * from "./types.slack.js";

View File

@@ -3,6 +3,49 @@ import { isSafeExecutableValue } from "../infra/exec-safety.js";
import { createAllowDenyChannelRulesSchema } from "./zod-schema.allowdeny.js";
import { sensitive } from "./zod-schema.sensitive.js";
const SECRET_REF_ID_PATTERN = /^[A-Za-z0-9_./:=-](?:[A-Za-z0-9_./:=~-]{0,127})$/;
export const SecretRefSchema = z
.object({
source: z.enum(["env", "file"]),
id: z
.string()
.regex(
SECRET_REF_ID_PATTERN,
"Secret reference id must match /^[A-Za-z0-9_./:=-](?:[A-Za-z0-9_./:=~-]{0,127})$/",
),
})
.strict();
export const SecretInputSchema = z.union([z.string(), SecretRefSchema]);
const SecretsEnvSourceSchema = z
.object({
type: z.literal("env").optional(),
})
.strict();
const SecretsFileSourceSchema = z
.object({
type: z.literal("sops"),
path: z.string().min(1),
timeoutMs: z.number().int().positive().max(120000).optional(),
})
.strict();
export const SecretsConfigSchema = z
.object({
sources: z
.object({
env: SecretsEnvSourceSchema.optional(),
file: SecretsFileSourceSchema.optional(),
})
.strict()
.optional(),
})
.strict()
.optional();
export const ModelApiSchema = z.union([
z.literal("openai-completions"),
z.literal("openai-responses"),
@@ -58,7 +101,7 @@ export const ModelDefinitionSchema = z
export const ModelProviderSchema = z
.object({
baseUrl: z.string().min(1),
apiKey: z.string().optional().register(sensitive),
apiKey: SecretInputSchema.optional().register(sensitive),
auth: z
.union([z.literal("api-key"), z.literal("aws-sdk"), z.literal("oauth"), z.literal("token")])
.optional(),

View File

@@ -25,6 +25,7 @@ import {
MarkdownConfigSchema,
MSTeamsReplyStyleSchema,
ProviderCommandsSchema,
SecretRefSchema,
ReplyToModeSchema,
RetryConfigSchema,
TtsConfigSchema,
@@ -519,7 +520,11 @@ export const GoogleChatAccountSchema = z
groupAllowFrom: z.array(z.union([z.string(), z.number()])).optional(),
groups: z.record(z.string(), GoogleChatGroupSchema.optional()).optional(),
defaultTo: z.string().optional(),
serviceAccount: z.union([z.string(), z.record(z.string(), z.unknown())]).optional(),
serviceAccount: z
.union([z.string(), z.record(z.string(), z.unknown()), SecretRefSchema])
.optional()
.register(sensitive),
serviceAccountRef: SecretRefSchema.optional().register(sensitive),
serviceAccountFile: z.string().optional(),
audienceType: z.enum(["app-url", "project-number"]).optional(),
audience: z.string().optional(),

View File

@@ -2,7 +2,12 @@ import { z } from "zod";
import { ToolsSchema } from "./zod-schema.agent-runtime.js";
import { AgentsSchema, AudioSchema, BindingsSchema, BroadcastSchema } from "./zod-schema.agents.js";
import { ApprovalsSchema } from "./zod-schema.approvals.js";
import { HexColorSchema, ModelsConfigSchema } from "./zod-schema.core.js";
import {
HexColorSchema,
ModelsConfigSchema,
SecretInputSchema,
SecretsConfigSchema,
} from "./zod-schema.core.js";
import { HookMappingSchema, HooksGmailSchema, InternalHooksSchema } from "./zod-schema.hooks.js";
import { InstallRecordShape } from "./zod-schema.installs.js";
import { ChannelsSchema } from "./zod-schema.providers.js";
@@ -72,18 +77,9 @@ const MemoryQmdLimitsSchema = z
})
.strict();
const MemoryQmdMcporterSchema = z
.object({
enabled: z.boolean().optional(),
serverName: z.string().optional(),
startDaemon: z.boolean().optional(),
})
.strict();
const MemoryQmdSchema = z
.object({
command: z.string().optional(),
mcporter: MemoryQmdMcporterSchema.optional(),
searchMode: z.union([z.literal("query"), z.literal("search"), z.literal("vsearch")]).optional(),
includeDefaultMemory: z.boolean().optional(),
paths: z.array(MemoryQmdPathSchema).optional(),
@@ -272,6 +268,7 @@ export const OpenClawSchema = z
})
.strict()
.optional(),
secrets: SecretsConfigSchema,
auth: z
.object({
profiles: z
@@ -621,7 +618,7 @@ export const OpenClawSchema = z
z
.object({
enabled: z.boolean().optional(),
apiKey: z.string().optional().register(sensitive),
apiKey: SecretInputSchema.optional().register(sensitive),
env: z.record(z.string(), z.string()).optional(),
config: z.record(z.string(), z.unknown()).optional(),
})

View File

@@ -13,18 +13,6 @@ describe("cron schedule", () => {
expect(next).toBe(Date.parse("2025-12-17T17:00:00.000Z"));
});
it("throws a clear error when cron expr is missing at runtime", () => {
const nowMs = Date.parse("2025-12-13T00:00:00.000Z");
expect(() =>
computeNextRunAtMs(
{
kind: "cron",
} as unknown as { kind: "cron"; expr: string; tz?: string },
nowMs,
),
).toThrow("invalid cron schedule: expr is required");
});
it("computes next run for every schedule", () => {
const anchor = Date.parse("2025-12-13T00:00:00.000Z");
const now = anchor + 10_000;

View File

@@ -41,11 +41,7 @@ export function computeNextRunAtMs(schedule: CronSchedule, nowMs: number): numbe
return anchor + steps * everyMs;
}
const exprSource = (schedule as { expr?: unknown }).expr;
if (typeof exprSource !== "string") {
throw new Error("invalid cron schedule: expr is required");
}
const expr = exprSource.trim();
const expr = schedule.expr.trim();
if (!expr) {
return undefined;
}

View File

@@ -11,22 +11,6 @@ const noopLogger = {
error: vi.fn(),
};
async function withTimeout<T>(promise: Promise<T>, timeoutMs: number, label: string): Promise<T> {
let timeout: NodeJS.Timeout | undefined;
try {
return await Promise.race([
promise,
new Promise<T>((_resolve, reject) => {
timeout = setTimeout(() => reject(new Error(`${label} timed out`)), timeoutMs);
}),
]);
} finally {
if (timeout) {
clearTimeout(timeout);
}
}
}
async function makeStorePath() {
const dir = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-cron-"));
return {
@@ -151,86 +135,4 @@ describe("CronService read ops while job is running", () => {
await store.cleanup();
}
});
it("keeps list and status responsive during startup catch-up runs", async () => {
const store = await makeStorePath();
const enqueueSystemEvent = vi.fn();
const requestHeartbeatNow = vi.fn();
const nowMs = Date.parse("2025-12-13T00:00:00.000Z");
await fs.mkdir(path.dirname(store.storePath), { recursive: true });
await fs.writeFile(
store.storePath,
JSON.stringify({
version: 1,
jobs: [
{
id: "startup-catchup",
name: "startup catch-up",
enabled: true,
createdAtMs: nowMs - 86_400_000,
updatedAtMs: nowMs - 86_400_000,
schedule: { kind: "at", at: new Date(nowMs - 60_000).toISOString() },
sessionTarget: "isolated",
wakeMode: "next-heartbeat",
payload: { kind: "agentTurn", message: "startup replay" },
delivery: { mode: "none" },
state: { nextRunAtMs: nowMs - 60_000 },
},
],
}),
"utf-8",
);
let resolveRun:
| ((value: { status: "ok" | "error" | "skipped"; summary?: string; error?: string }) => void)
| undefined;
let resolveRunStarted: (() => void) | undefined;
const runStarted = new Promise<void>((resolve) => {
resolveRunStarted = resolve;
});
const runIsolatedAgentJob = vi.fn(async () => {
resolveRunStarted?.();
return await new Promise<{
status: "ok" | "error" | "skipped";
summary?: string;
error?: string;
}>((resolve) => {
resolveRun = resolve;
});
});
const cron = new CronService({
storePath: store.storePath,
cronEnabled: true,
log: noopLogger,
nowMs: () => nowMs,
enqueueSystemEvent,
requestHeartbeatNow,
runIsolatedAgentJob,
});
try {
const startPromise = cron.start();
await runStarted;
await expect(
withTimeout(cron.list({ includeDisabled: true }), 300, "cron.list during startup"),
).resolves.toBeTypeOf("object");
await expect(withTimeout(cron.status(), 300, "cron.status during startup")).resolves.toEqual(
expect.objectContaining({ enabled: true, storePath: store.storePath }),
);
resolveRun?.({ status: "ok", summary: "done" });
await startPromise;
const jobs = await cron.list({ includeDisabled: true });
expect(jobs[0]?.state.lastStatus).toBe("ok");
expect(jobs[0]?.state.runningAtMs).toBeUndefined();
} finally {
cron.stop();
await store.cleanup();
}
});
});

View File

@@ -186,19 +186,4 @@ describe("cron schedule error isolation", () => {
expect(badJob.state.lastError).toMatch(/^schedule error:/);
expect(badJob.state.lastError).toBeTruthy();
});
it("records a clear schedule error when cron expr is missing", () => {
const badJob = createJob({
id: "missing-expr",
name: "Missing Expr",
schedule: { kind: "cron" } as unknown as CronJob["schedule"],
});
const state = createMockState([badJob]);
recomputeNextRuns(state);
expect(badJob.state.lastError).toContain("invalid cron schedule: expr is required");
expect(badJob.state.lastError).not.toContain("Cannot read properties of undefined");
expect(badJob.state.scheduleErrorCount).toBe(1);
});
});

View File

@@ -28,15 +28,14 @@ async function ensureLoadedForRead(state: CronServiceState) {
}
export async function start(state: CronServiceState) {
if (!state.deps.cronEnabled) {
state.deps.log.info({ enabled: false }, "cron: disabled");
return;
}
const startupInterruptedJobIds = new Set<string>();
await locked(state, async () => {
if (!state.deps.cronEnabled) {
state.deps.log.info({ enabled: false }, "cron: disabled");
return;
}
await ensureLoaded(state, { skipRecompute: true });
const jobs = state.store?.jobs ?? [];
const startupInterruptedJobIds = new Set<string>();
for (const job of jobs) {
if (typeof job.state.runningAtMs === "number") {
state.deps.log.warn(
@@ -47,13 +46,7 @@ export async function start(state: CronServiceState) {
startupInterruptedJobIds.add(job.id);
}
}
await persist(state);
});
await runMissedJobs(state, { skipJobIds: startupInterruptedJobIds });
await locked(state, async () => {
await ensureLoaded(state, { forceReload: true, skipRecompute: true });
await runMissedJobs(state, { skipJobIds: startupInterruptedJobIds });
recomputeNextRuns(state);
await persist(state);
armTimer(state);

View File

@@ -458,97 +458,22 @@ export async function runMissedJobs(
state: CronServiceState,
opts?: { skipJobIds?: ReadonlySet<string> },
) {
const startupCandidates = await locked(state, async () => {
await ensureLoaded(state, { skipRecompute: true });
if (!state.store) {
return [] as Array<{ jobId: string; job: CronJob }>;
}
const now = state.deps.nowMs();
const skipJobIds = opts?.skipJobIds;
const missed = collectRunnableJobs(state, now, { skipJobIds, skipAtIfAlreadyRan: true });
if (missed.length === 0) {
return [] as Array<{ jobId: string; job: CronJob }>;
}
if (!state.store) {
return;
}
const now = state.deps.nowMs();
const skipJobIds = opts?.skipJobIds;
const missed = collectRunnableJobs(state, now, { skipJobIds, skipAtIfAlreadyRan: true });
if (missed.length > 0) {
state.deps.log.info(
{ count: missed.length, jobIds: missed.map((j) => j.id) },
"cron: running missed jobs after restart",
);
for (const job of missed) {
job.state.runningAtMs = now;
job.state.lastError = undefined;
}
await persist(state);
return missed.map((job) => ({ jobId: job.id, job }));
});
if (startupCandidates.length === 0) {
return;
}
const outcomes: Array<TimedCronRunOutcome> = [];
for (const candidate of startupCandidates) {
const startedAt = state.deps.nowMs();
emit(state, { jobId: candidate.job.id, action: "started", runAtMs: startedAt });
try {
const result = await executeJobCore(state, candidate.job);
outcomes.push({
jobId: candidate.jobId,
status: result.status,
error: result.error,
summary: result.summary,
delivered: result.delivered,
sessionId: result.sessionId,
sessionKey: result.sessionKey,
model: result.model,
provider: result.provider,
usage: result.usage,
startedAt,
endedAt: state.deps.nowMs(),
});
} catch (err) {
outcomes.push({
jobId: candidate.jobId,
status: "error",
error: String(err),
startedAt,
endedAt: state.deps.nowMs(),
});
await executeJob(state, job, now, { forced: false });
}
}
await locked(state, async () => {
await ensureLoaded(state, { forceReload: true, skipRecompute: true });
if (!state.store) {
return;
}
for (const result of outcomes) {
const job = state.store.jobs.find((entry) => entry.id === result.jobId);
if (!job) {
continue;
}
const shouldDelete = applyJobResult(state, job, {
status: result.status,
error: result.error,
delivered: result.delivered,
startedAt: result.startedAt,
endedAt: result.endedAt,
});
emitJobFinished(state, job, result, result.startedAt);
if (shouldDelete) {
state.store.jobs = state.store.jobs.filter((entry) => entry.id !== job.id);
emit(state, { jobId: job.id, action: "removed" });
}
}
// Preserve any new past-due nextRunAtMs values that became due while
// startup catch-up was running. They should execute on a future tick
// instead of being silently advanced.
recomputeNextRunsForMaintenance(state);
await persist(state);
});
}
export async function runDueJobs(state: CronServiceState) {

View File

@@ -33,13 +33,4 @@ describe("cron stagger helpers", () => {
expect(resolveCronStaggerMs({ kind: "cron", expr: "0 * * * *", staggerMs: 0 })).toBe(0);
expect(resolveCronStaggerMs({ kind: "cron", expr: "15 * * * *" })).toBe(0);
});
it("handles missing runtime expr values without throwing", () => {
expect(() =>
resolveCronStaggerMs({ kind: "cron" } as unknown as { kind: "cron"; expr: string }),
).not.toThrow();
expect(
resolveCronStaggerMs({ kind: "cron" } as unknown as { kind: "cron"; expr: string }),
).toBe(0);
});
});

View File

@@ -41,7 +41,5 @@ export function resolveCronStaggerMs(schedule: Extract<CronSchedule, { kind: "cr
if (explicit !== undefined) {
return explicit;
}
const expr = (schedule as { expr?: unknown }).expr;
const cronExpr = typeof expr === "string" ? expr : "";
return resolveDefaultCronStaggerMs(cronExpr) ?? 0;
return resolveDefaultCronStaggerMs(schedule.expr) ?? 0;
}

View File

@@ -23,48 +23,6 @@ describe("diffConfigPaths", () => {
const paths = diffConfigPaths(prev, next);
expect(paths).toContain("messages.groupChat.mentionPatterns");
});
it("does not report unchanged arrays of objects as changed", () => {
const prev = {
memory: {
qmd: {
paths: [{ path: "~/docs", pattern: "**/*.md", name: "docs" }],
scope: {
rules: [{ when: { channel: "slack" }, include: ["docs"] }],
},
},
},
};
const next = {
memory: {
qmd: {
paths: [{ path: "~/docs", pattern: "**/*.md", name: "docs" }],
scope: {
rules: [{ when: { channel: "slack" }, include: ["docs"] }],
},
},
},
};
expect(diffConfigPaths(prev, next)).toEqual([]);
});
it("reports changed arrays of objects", () => {
const prev = {
memory: {
qmd: {
paths: [{ path: "~/docs", pattern: "**/*.md", name: "docs" }],
},
},
};
const next = {
memory: {
qmd: {
paths: [{ path: "~/docs", pattern: "**/*.txt", name: "docs" }],
},
},
};
expect(diffConfigPaths(prev, next)).toContain("memory.qmd.paths");
});
});
describe("buildGatewayReloadPlan", () => {

View File

@@ -1,4 +1,3 @@
import { isDeepStrictEqual } from "node:util";
import chokidar from "chokidar";
import { type ChannelId, listChannelPlugins } from "../channels/plugins/index.js";
import type { OpenClawConfig, ConfigFileSnapshot, GatewayReloadMode } from "../config/config.js";
@@ -151,9 +150,7 @@ export function diffConfigPaths(prev: unknown, next: unknown, prefix = ""): stri
return paths;
}
if (Array.isArray(prev) && Array.isArray(next)) {
// Arrays can contain object entries (for example memory.qmd.paths/scope.rules);
// compare structurally so identical values are not reported as changed.
if (isDeepStrictEqual(prev, next)) {
if (prev.length === next.length && prev.every((val, idx) => val === next[idx])) {
return [];
}
}
@@ -253,7 +250,7 @@ export function startGatewayConfigReloader(opts: {
initialConfig: OpenClawConfig;
readSnapshot: () => Promise<ConfigFileSnapshot>;
onHotReload: (plan: GatewayReloadPlan, nextConfig: OpenClawConfig) => Promise<void>;
onRestart: (plan: GatewayReloadPlan, nextConfig: OpenClawConfig) => void;
onRestart: (plan: GatewayReloadPlan, nextConfig: OpenClawConfig) => void | Promise<void>;
log: {
info: (msg: string) => void;
warn: (msg: string) => void;
@@ -304,42 +301,53 @@ export function startGatewayConfigReloader(opts: {
}
const nextConfig = snapshot.config;
const changedPaths = diffConfigPaths(currentConfig, nextConfig);
currentConfig = nextConfig;
settings = resolveGatewayReloadSettings(nextConfig);
const nextSettings = resolveGatewayReloadSettings(nextConfig);
if (changedPaths.length === 0) {
currentConfig = nextConfig;
settings = nextSettings;
return;
}
opts.log.info(`config change detected; evaluating reload (${changedPaths.join(", ")})`);
const plan = buildGatewayReloadPlan(changedPaths);
if (settings.mode === "off") {
if (nextSettings.mode === "off") {
opts.log.info("config reload disabled (gateway.reload.mode=off)");
currentConfig = nextConfig;
settings = nextSettings;
return;
}
if (settings.mode === "restart") {
if (nextSettings.mode === "restart") {
if (!restartQueued) {
restartQueued = true;
opts.onRestart(plan, nextConfig);
await opts.onRestart(plan, nextConfig);
}
currentConfig = nextConfig;
settings = nextSettings;
return;
}
if (plan.restartGateway) {
if (settings.mode === "hot") {
if (nextSettings.mode === "hot") {
opts.log.warn(
`config reload requires gateway restart; hot mode ignoring (${plan.restartReasons.join(
", ",
)})`,
);
currentConfig = nextConfig;
settings = nextSettings;
return;
}
if (!restartQueued) {
restartQueued = true;
opts.onRestart(plan, nextConfig);
await opts.onRestart(plan, nextConfig);
}
currentConfig = nextConfig;
settings = nextSettings;
return;
}
await opts.onHotReload(plan, nextConfig);
currentConfig = nextConfig;
settings = nextSettings;
} catch (err) {
opts.log.error(`config reload failed: ${String(err)}`);
} finally {

View File

@@ -97,6 +97,7 @@ const METHOD_SCOPE_GROUPS: Record<OperatorScope, readonly string[]> = {
"agents.delete",
"skills.install",
"skills.update",
"secrets.reload",
"cron.add",
"cron.update",
"cron.remove",

View File

@@ -48,6 +48,7 @@ const BASE_METHODS = [
"update.run",
"voicewake.get",
"voicewake.set",
"secrets.reload",
"sessions.list",
"sessions.preview",
"sessions.patch",

View File

@@ -1,6 +1,3 @@
import { DEFAULT_PROVIDER } from "../../agents/defaults.js";
import { buildAllowedModelSet } from "../../agents/model-selection.js";
import { loadConfig } from "../../config/config.js";
import {
ErrorCodes,
errorShape,
@@ -23,14 +20,7 @@ export const modelsHandlers: GatewayRequestHandlers = {
return;
}
try {
const catalog = await context.loadGatewayModelCatalog();
const cfg = loadConfig();
const { allowedCatalog } = buildAllowedModelSet({
cfg,
catalog,
defaultProvider: DEFAULT_PROVIDER,
});
const models = allowedCatalog.length > 0 ? allowedCatalog : catalog;
const models = await context.loadGatewayModelCatalog();
respond(true, { models }, undefined);
} catch (err) {
respond(false, undefined, errorShape(ErrorCodes.UNAVAILABLE, String(err)));

View File

@@ -0,0 +1,43 @@
import { describe, expect, it, vi } from "vitest";
import { createSecretsHandlers } from "./secrets.js";
describe("secrets handlers", () => {
it("responds with warning count on successful reload", async () => {
const handlers = createSecretsHandlers({
reloadSecrets: vi.fn().mockResolvedValue({ warningCount: 2 }),
});
const respond = vi.fn();
await handlers["secrets.reload"]({
req: { type: "req", id: "1", method: "secrets.reload" },
params: {},
client: null,
isWebchatConnect: () => false,
respond,
context: {} as never,
});
expect(respond).toHaveBeenCalledWith(true, { ok: true, warningCount: 2 });
});
it("returns unavailable when reload fails", async () => {
const handlers = createSecretsHandlers({
reloadSecrets: vi.fn().mockRejectedValue(new Error("reload failed")),
});
const respond = vi.fn();
await handlers["secrets.reload"]({
req: { type: "req", id: "1", method: "secrets.reload" },
params: {},
client: null,
isWebchatConnect: () => false,
respond,
context: {} as never,
});
expect(respond).toHaveBeenCalledWith(
false,
undefined,
expect.objectContaining({
code: "UNAVAILABLE",
message: "Error: reload failed",
}),
);
});
});

View File

@@ -0,0 +1,17 @@
import { ErrorCodes, errorShape } from "../protocol/index.js";
import type { GatewayRequestHandlers } from "./types.js";
export function createSecretsHandlers(params: {
reloadSecrets: () => Promise<{ warningCount: number }>;
}): GatewayRequestHandlers {
return {
"secrets.reload": async ({ respond }) => {
try {
const result = await params.reloadSecrets();
respond(true, { ok: true, warningCount: result.warningCount });
} catch (err) {
respond(false, undefined, errorShape(ErrorCodes.UNAVAILABLE, String(err)));
}
},
};
}

View File

@@ -11,6 +11,7 @@ import { createDefaultDeps } from "../cli/deps.js";
import { isRestartEnabled } from "../config/commands.js";
import {
CONFIG_PATH,
type OpenClawConfig,
isNixMode,
loadConfig,
migrateLegacyConfig,
@@ -45,6 +46,12 @@ import { createEmptyPluginRegistry } from "../plugins/registry.js";
import type { PluginServicesHandle } from "../plugins/services.js";
import { getTotalQueueSize } from "../process/command-queue.js";
import type { RuntimeEnv } from "../runtime.js";
import {
activateSecretsRuntimeSnapshot,
clearSecretsRuntimeSnapshot,
getActiveSecretsRuntimeSnapshot,
prepareSecretsRuntimeSnapshot,
} from "../secrets/runtime.js";
import { runOnboardingWizard } from "../wizard/onboarding.js";
import { createAuthRateLimiter, type AuthRateLimiter } from "./auth-rate-limit.js";
import { startChannelHealthMonitor } from "./channel-health-monitor.js";
@@ -68,6 +75,7 @@ import { GATEWAY_EVENTS, listGatewayMethods } from "./server-methods-list.js";
import { coreGatewayHandlers } from "./server-methods.js";
import { createExecApprovalHandlers } from "./server-methods/exec-approval.js";
import { safeParseJson } from "./server-methods/nodes.helpers.js";
import { createSecretsHandlers } from "./server-methods/secrets.js";
import { hasConnectedMobileNode } from "./server-mobile-nodes.js";
import { loadGatewayModelCatalog } from "./server-model-catalog.js";
import { createNodeSubscriptionManager } from "./server-node-subscriptions.js";
@@ -107,6 +115,7 @@ const logReload = log.child("reload");
const logHooks = log.child("hooks");
const logPlugins = log.child("plugins");
const logWsControl = log.child("ws");
const logSecrets = log.child("secrets");
const gatewayRuntime = runtimeForLogger(log);
const canvasRuntime = runtimeForLogger(logCanvas);
@@ -233,7 +242,63 @@ export async function startGatewayServer(
}
}
let cfgAtStart = loadConfig();
let secretsDegraded = false;
const activateRuntimeSecrets = async (
config: OpenClawConfig,
params: { reason: "startup" | "reload" | "restart-check"; activate: boolean },
) => {
try {
const prepared = await prepareSecretsRuntimeSnapshot({ config });
if (params.activate) {
activateSecretsRuntimeSnapshot(prepared);
}
for (const warning of prepared.warnings) {
logSecrets.warn(`[${warning.code}] ${warning.message}`);
}
if (secretsDegraded) {
logSecrets.info(
"[SECRETS_RELOADER_RECOVERED] Secret resolution recovered; runtime remained on last-known-good during the outage.",
);
}
secretsDegraded = false;
return prepared;
} catch (err) {
const details = String(err);
if (!secretsDegraded) {
logSecrets.error(`[SECRETS_RELOADER_DEGRADED] ${details}`);
} else {
logSecrets.warn(`[SECRETS_RELOADER_DEGRADED] ${details}`);
}
secretsDegraded = true;
if (params.reason === "startup") {
throw new Error(`Startup failed: required secrets are unavailable. ${details}`, {
cause: err,
});
}
throw err;
}
};
// Fail fast before startup if required refs are unresolved.
let cfgAtStart: OpenClawConfig;
{
const freshSnapshot = await readConfigFileSnapshot();
if (!freshSnapshot.valid) {
const issues =
freshSnapshot.issues.length > 0
? freshSnapshot.issues
.map((issue) => `${issue.path || "<root>"}: ${issue.message}`)
.join("\n")
: "Unknown validation issue.";
throw new Error(`Invalid config at ${freshSnapshot.path}.\n${issues}`);
}
await activateRuntimeSecrets(freshSnapshot.config, {
reason: "startup",
activate: false,
});
}
cfgAtStart = loadConfig();
const authBootstrap = await ensureGatewayStartupAuth({
cfg: cfgAtStart,
env: process.env,
@@ -253,6 +318,12 @@ export async function startGatewayServer(
);
}
}
cfgAtStart = (
await activateRuntimeSecrets(cfgAtStart, {
reason: "startup",
activate: true,
})
).config;
const diagnosticsEnabled = isDiagnosticsEnabled(cfgAtStart);
if (diagnosticsEnabled) {
startDiagnosticHeartbeat();
@@ -560,6 +631,19 @@ export async function startGatewayServer(
const execApprovalHandlers = createExecApprovalHandlers(execApprovalManager, {
forwarder: execApprovalForwarder,
});
const secretsHandlers = createSecretsHandlers({
reloadSecrets: async () => {
const active = getActiveSecretsRuntimeSnapshot();
if (!active) {
throw new Error("Secrets runtime snapshot is not active.");
}
const prepared = await activateRuntimeSecrets(active.sourceConfig, {
reason: "reload",
activate: true,
});
return { warningCount: prepared.warnings.length };
},
});
const canvasHostServerPort = (canvasHostServer as CanvasHostServer | null)?.port;
@@ -580,6 +664,7 @@ export async function startGatewayServer(
extraHandlers: {
...pluginRegistry.gatewayHandlers,
...execApprovalHandlers,
...secretsHandlers,
},
broadcast,
context: {
@@ -710,8 +795,27 @@ export async function startGatewayServer(
return startGatewayConfigReloader({
initialConfig: cfgAtStart,
readSnapshot: readConfigFileSnapshot,
onHotReload: applyHotReload,
onRestart: requestGatewayRestart,
onHotReload: async (plan, nextConfig) => {
const previousSnapshot = getActiveSecretsRuntimeSnapshot();
const prepared = await activateRuntimeSecrets(nextConfig, {
reason: "reload",
activate: true,
});
try {
await applyHotReload(plan, prepared.config);
} catch (err) {
if (previousSnapshot) {
activateSecretsRuntimeSnapshot(previousSnapshot);
} else {
clearSecretsRuntimeSnapshot();
}
throw err;
}
},
onRestart: async (plan, nextConfig) => {
await activateRuntimeSecrets(nextConfig, { reason: "restart-check", activate: false });
requestGatewayRestart(plan, nextConfig);
},
log: {
info: (msg) => logReload.info(msg),
warn: (msg) => logReload.warn(msg),
@@ -764,6 +868,7 @@ export async function startGatewayServer(
skillsChangeUnsub();
authRateLimiter?.dispose();
channelHealthMonitor?.stop();
clearSecretsRuntimeSnapshot();
await close(opts);
},
};

View File

@@ -5,7 +5,6 @@ import { afterAll, beforeAll, describe, expect, test } from "vitest";
import { WebSocket } from "ws";
import { getChannelPlugin } from "../channels/plugins/index.js";
import type { ChannelOutboundAdapter } from "../channels/plugins/types.js";
import { clearConfigCache } from "../config/config.js";
import { resolveCanvasHostUrl } from "../infra/canvas-host-url.js";
import { GatewayLockError } from "../infra/gateway-lock.js";
import { getActivePluginRegistry, setActivePluginRegistry } from "../plugins/runtime.js";
@@ -252,99 +251,6 @@ describe("gateway server models + voicewake", () => {
expect(piSdkMock.discoverCalls).toBe(1);
});
test("models.list filters to allowlisted configured models by default", async () => {
const configPath = process.env.OPENCLAW_CONFIG_PATH;
if (!configPath) {
throw new Error("Missing OPENCLAW_CONFIG_PATH");
}
let previousConfig: string | undefined;
try {
previousConfig = await fs.readFile(configPath, "utf-8");
} catch (err) {
const code = (err as NodeJS.ErrnoException | undefined)?.code;
if (code !== "ENOENT") {
throw err;
}
}
try {
await fs.mkdir(path.dirname(configPath), { recursive: true });
await fs.writeFile(
configPath,
JSON.stringify(
{
agents: {
defaults: {
model: { primary: "openai/gpt-test-z" },
models: {
"openai/gpt-test-z": {},
"anthropic/claude-test-a": {},
},
},
},
},
null,
2,
),
"utf-8",
);
clearConfigCache();
piSdkMock.enabled = true;
piSdkMock.models = [
{ id: "gpt-test-z", provider: "openai", contextWindow: 0 },
{
id: "gpt-test-a",
name: "A-Model",
provider: "openai",
contextWindow: 8000,
},
{
id: "claude-test-b",
name: "B-Model",
provider: "anthropic",
contextWindow: 1000,
},
{
id: "claude-test-a",
name: "A-Model",
provider: "anthropic",
contextWindow: 200_000,
},
];
const res = await rpcReq<{
models: Array<{
id: string;
name: string;
provider: string;
contextWindow?: number;
}>;
}>(ws, "models.list");
expect(res.ok).toBe(true);
expect(res.payload?.models).toEqual([
{
id: "claude-test-a",
name: "A-Model",
provider: "anthropic",
contextWindow: 200_000,
},
{
id: "gpt-test-z",
name: "gpt-test-z",
provider: "openai",
},
]);
} finally {
if (previousConfig === undefined) {
await fs.rm(configPath, { force: true });
} else {
await fs.writeFile(configPath, previousConfig, "utf-8");
}
clearConfigCache();
}
});
test("models.list rejects unknown params", async () => {
piSdkMock.enabled = true;
piSdkMock.models = [{ id: "gpt-test-a", name: "A", provider: "openai" }];

View File

@@ -122,26 +122,6 @@ describe("device pairing tokens", () => {
expect(paired?.tokens?.operator?.scopes).toEqual(["operator.read"]);
});
test("preserves existing token scopes when approving a repair without requested scopes", async () => {
const baseDir = await mkdtemp(join(tmpdir(), "openclaw-device-pairing-"));
await setupPairedOperatorDevice(baseDir, ["operator.admin"]);
const repair = await requestDevicePairing(
{
deviceId: "device-1",
publicKey: "public-key-1",
role: "operator",
},
baseDir,
);
await approveDevicePairing(repair.request.requestId, baseDir);
const paired = await getPairedDevice("device-1", baseDir);
expect(paired?.scopes).toEqual(["operator.admin"]);
expect(paired?.approvedScopes).toEqual(["operator.admin"]);
expect(paired?.tokens?.operator?.scopes).toEqual(["operator.admin"]);
});
test("rejects scope escalation when rotating a token and leaves state unchanged", async () => {
const baseDir = await mkdtemp(join(tmpdir(), "openclaw-device-pairing-"));
await setupPairedOperatorDevice(baseDir, ["operator.read"]);
@@ -188,7 +168,7 @@ describe("device pairing tokens", () => {
expect(mismatch.reason).toBe("token-mismatch");
});
test("accepts operator.read/operator.write requests with an operator.admin token scope", async () => {
test("accepts operator.read requests with an operator.admin token scope", async () => {
const baseDir = await mkdtemp(join(tmpdir(), "openclaw-device-pairing-"));
await setupPairedOperatorDevice(baseDir, ["operator.admin"]);
const paired = await getPairedDevice("device-1", baseDir);
@@ -203,14 +183,14 @@ describe("device pairing tokens", () => {
});
expect(readOk.ok).toBe(true);
const writeOk = await verifyDeviceToken({
const writeMismatch = await verifyDeviceToken({
deviceId: "device-1",
token,
role: "operator",
scopes: ["operator.write"],
baseDir,
});
expect(writeOk.ok).toBe(true);
expect(writeMismatch).toEqual({ ok: false, reason: "scope-mismatch" });
});
test("treats multibyte same-length token input as mismatch without throwing", async () => {

View File

@@ -332,17 +332,8 @@ export async function approveDevicePairing(
const tokens = existing?.tokens ? { ...existing.tokens } : {};
const roleForToken = normalizeRole(pending.role);
if (roleForToken) {
const nextScopes = normalizeDeviceAuthScopes(pending.scopes);
const existingToken = tokens[roleForToken];
const requestedScopes = normalizeDeviceAuthScopes(pending.scopes);
const nextScopes =
requestedScopes.length > 0
? requestedScopes
: normalizeDeviceAuthScopes(
existingToken?.scopes ??
approvedScopes ??
existing?.approvedScopes ??
existing?.scopes,
);
const now = Date.now();
tokens[roleForToken] = {
token: newToken(),

View File

@@ -17,6 +17,7 @@ import {
validateSafeBinArgv,
} from "./exec-safe-bin-policy.js";
import { isTrustedSafeBinPath } from "./exec-safe-bin-trust.js";
export function normalizeSafeBins(entries?: string[]): Set<string> {
if (!Array.isArray(entries)) {
return new Set();

View File

@@ -196,55 +196,6 @@ describe("gateway lock", () => {
staleSpy.mockRestore();
});
it("keeps lock when fs.stat fails until payload is stale", async () => {
vi.useRealTimers();
const env = await makeEnv();
const { lockPath, configPath } = resolveLockPath(env);
const payload = createLockPayload({ configPath, startTime: 111 });
await fs.writeFile(lockPath, JSON.stringify(payload), "utf8");
const procSpy = mockProcStatRead({
onProcRead: () => {
throw new Error("EACCES");
},
});
const statSpy = vi
.spyOn(fs, "stat")
.mockRejectedValue(Object.assign(new Error("EPERM"), { code: "EPERM" }));
const pending = acquireForTest(env, {
timeoutMs: 20,
staleMs: 10_000,
platform: "linux",
});
await expect(pending).rejects.toBeInstanceOf(GatewayLockError);
procSpy.mockRestore();
const stalePayload = createLockPayload({
configPath,
startTime: 111,
createdAt: new Date(0).toISOString(),
});
await fs.writeFile(lockPath, JSON.stringify(stalePayload), "utf8");
const staleProcSpy = mockProcStatRead({
onProcRead: () => {
throw new Error("EACCES");
},
});
const lock = await acquireForTest(env, {
staleMs: 1,
platform: "linux",
});
expect(lock).not.toBeNull();
await lock?.release();
staleProcSpy.mockRestore();
statSpy.mockRestore();
});
it("returns null when multi-gateway override is enabled", async () => {
const env = await makeEnv();
const lock = await acquireGatewayLock({

View File

@@ -231,11 +231,7 @@ export async function acquireGatewayLock(
const st = await fs.stat(lockPath);
stale = Date.now() - st.mtimeMs > staleMs;
} catch {
// On Windows or locked filesystems we may be unable to stat the
// lock file even though the existing gateway is still healthy.
// Treat the lock as non-stale so we keep waiting instead of
// forcefully removing another gateway's lock.
stale = false;
stale = true;
}
}
if (stale) {

View File

@@ -8,7 +8,6 @@ import type {
MemoryCitationsMode,
MemoryQmdConfig,
MemoryQmdIndexPath,
MemoryQmdMcporterConfig,
MemoryQmdSearchMode,
} from "../config/types.memory.js";
import { resolveUserPath } from "../utils.js";
@@ -51,15 +50,8 @@ export type ResolvedQmdSessionConfig = {
retentionDays?: number;
};
export type ResolvedQmdMcporterConfig = {
enabled: boolean;
serverName: string;
startDaemon: boolean;
};
export type ResolvedQmdConfig = {
command: string;
mcporter: ResolvedQmdMcporterConfig;
searchMode: MemoryQmdSearchMode;
collections: ResolvedQmdCollection[];
sessions: ResolvedQmdSessionConfig;
@@ -87,12 +79,6 @@ const DEFAULT_QMD_LIMITS: ResolvedQmdLimitsConfig = {
maxInjectedChars: 4_000,
timeoutMs: DEFAULT_QMD_TIMEOUT_MS,
};
const DEFAULT_QMD_MCPORTER: ResolvedQmdMcporterConfig = {
enabled: false,
serverName: "qmd",
startDaemon: true,
};
const DEFAULT_QMD_SCOPE: SessionSendPolicyConfig = {
default: "deny",
rules: [
@@ -251,27 +237,6 @@ function resolveCustomPaths(
return collections;
}
function resolveMcporterConfig(raw?: MemoryQmdMcporterConfig): ResolvedQmdMcporterConfig {
const parsed: ResolvedQmdMcporterConfig = { ...DEFAULT_QMD_MCPORTER };
if (!raw) {
return parsed;
}
if (raw.enabled !== undefined) {
parsed.enabled = raw.enabled;
}
if (typeof raw.serverName === "string" && raw.serverName.trim()) {
parsed.serverName = raw.serverName.trim();
}
if (raw.startDaemon !== undefined) {
parsed.startDaemon = raw.startDaemon;
}
// When enabled, default startDaemon to true.
if (parsed.enabled && raw.startDaemon === undefined) {
parsed.startDaemon = true;
}
return parsed;
}
function resolveDefaultCollections(
include: boolean,
workspaceDir: string,
@@ -318,7 +283,6 @@ export function resolveMemoryBackendConfig(params: {
const command = parsedCommand?.[0] || rawCommand.split(/\s+/)[0] || "qmd";
const resolved: ResolvedQmdConfig = {
command,
mcporter: resolveMcporterConfig(qmdCfg?.mcporter),
searchMode: resolveSearchMode(qmdCfg?.searchMode),
collections,
includeDefaultMemory,

View File

@@ -185,9 +185,7 @@ export async function createEmbeddingProvider(
continue;
}
// Non-auth errors (e.g., network) are still fatal
const wrapped = new Error(message) as Error & { cause?: unknown };
wrapped.cause = err;
throw wrapped;
throw new Error(message, { cause: err });
}
}
@@ -230,9 +228,7 @@ export async function createEmbeddingProvider(
};
}
// Non-auth errors are still fatal
const wrapped = new Error(combinedReason) as Error & { cause?: unknown };
wrapped.cause = fallbackErr;
throw wrapped;
throw new Error(combinedReason, { cause: fallbackErr });
}
}
// No fallback configured - check if we should degrade to FTS-only
@@ -243,9 +239,7 @@ export async function createEmbeddingProvider(
providerUnavailableReason: reason,
};
}
const wrapped = new Error(reason) as Error & { cause?: unknown };
wrapped.cause = primaryErr;
throw wrapped;
throw new Error(reason, { cause: primaryErr });
}
}

View File

@@ -148,8 +148,6 @@ describe("QmdMemoryManager", () => {
afterEach(async () => {
vi.useRealTimers();
delete process.env.OPENCLAW_STATE_DIR;
delete (globalThis as Record<string, unknown>).__openclawMcporterDaemonStart;
delete (globalThis as Record<string, unknown>).__openclawMcporterColdStartWarned;
});
it("debounces back-to-back sync calls", async () => {
@@ -912,171 +910,6 @@ describe("QmdMemoryManager", () => {
await manager.close();
});
it("runs qmd searches via mcporter and warns when startDaemon=false", async () => {
cfg = {
...cfg,
memory: {
backend: "qmd",
qmd: {
includeDefaultMemory: false,
update: { interval: "0s", debounceMs: 60_000, onBoot: false },
paths: [{ path: workspaceDir, pattern: "**/*.md", name: "workspace" }],
mcporter: { enabled: true, serverName: "qmd", startDaemon: false },
},
},
} as OpenClawConfig;
spawnMock.mockImplementation((cmd: string, args: string[]) => {
const child = createMockChild({ autoClose: false });
if (cmd === "mcporter" && args[0] === "call") {
emitAndClose(child, "stdout", JSON.stringify({ results: [] }));
return child;
}
emitAndClose(child, "stdout", "[]");
return child;
});
const { manager } = await createManager();
logWarnMock.mockClear();
await expect(
manager.search("hello", { sessionKey: "agent:main:slack:dm:u123" }),
).resolves.toEqual([]);
const mcporterCalls = spawnMock.mock.calls.filter((call: unknown[]) => call[0] === "mcporter");
expect(mcporterCalls.length).toBeGreaterThan(0);
expect(mcporterCalls.some((call: unknown[]) => (call[1] as string[])[0] === "daemon")).toBe(
false,
);
expect(logWarnMock).toHaveBeenCalledWith(expect.stringContaining("cold-start"));
await manager.close();
});
it("passes manager-scoped XDG env to mcporter commands", async () => {
cfg = {
...cfg,
memory: {
backend: "qmd",
qmd: {
includeDefaultMemory: false,
update: { interval: "0s", debounceMs: 60_000, onBoot: false },
paths: [{ path: workspaceDir, pattern: "**/*.md", name: "workspace" }],
mcporter: { enabled: true, serverName: "qmd", startDaemon: false },
},
},
} as OpenClawConfig;
spawnMock.mockImplementation((cmd: string, args: string[]) => {
const child = createMockChild({ autoClose: false });
if (cmd === "mcporter" && args[0] === "call") {
emitAndClose(child, "stdout", JSON.stringify({ results: [] }));
return child;
}
emitAndClose(child, "stdout", "[]");
return child;
});
const { manager } = await createManager();
await manager.search("hello", { sessionKey: "agent:main:slack:dm:u123" });
const mcporterCall = spawnMock.mock.calls.find(
(call: unknown[]) => call[0] === "mcporter" && (call[1] as string[])[0] === "call",
);
expect(mcporterCall).toBeDefined();
const spawnOpts = mcporterCall?.[2] as { env?: NodeJS.ProcessEnv } | undefined;
const normalizePath = (value?: string) => value?.replace(/\\/g, "/");
expect(normalizePath(spawnOpts?.env?.XDG_CONFIG_HOME)).toContain("/agents/main/qmd/xdg-config");
expect(normalizePath(spawnOpts?.env?.XDG_CACHE_HOME)).toContain("/agents/main/qmd/xdg-cache");
await manager.close();
});
it("retries mcporter daemon start after a failure", async () => {
cfg = {
...cfg,
memory: {
backend: "qmd",
qmd: {
includeDefaultMemory: false,
update: { interval: "0s", debounceMs: 60_000, onBoot: false },
paths: [{ path: workspaceDir, pattern: "**/*.md", name: "workspace" }],
mcporter: { enabled: true, serverName: "qmd", startDaemon: true },
},
},
} as OpenClawConfig;
let daemonAttempts = 0;
spawnMock.mockImplementation((cmd: string, args: string[]) => {
const child = createMockChild({ autoClose: false });
if (cmd === "mcporter" && args[0] === "daemon") {
daemonAttempts += 1;
if (daemonAttempts === 1) {
emitAndClose(child, "stderr", "failed", 1);
} else {
emitAndClose(child, "stdout", "");
}
return child;
}
if (cmd === "mcporter" && args[0] === "call") {
emitAndClose(child, "stdout", JSON.stringify({ results: [] }));
return child;
}
emitAndClose(child, "stdout", "[]");
return child;
});
const { manager } = await createManager();
await manager.search("one", { sessionKey: "agent:main:slack:dm:u123" });
await manager.search("two", { sessionKey: "agent:main:slack:dm:u123" });
expect(daemonAttempts).toBe(2);
await manager.close();
});
it("starts the mcporter daemon only once when enabled", async () => {
cfg = {
...cfg,
memory: {
backend: "qmd",
qmd: {
includeDefaultMemory: false,
update: { interval: "0s", debounceMs: 60_000, onBoot: false },
paths: [{ path: workspaceDir, pattern: "**/*.md", name: "workspace" }],
mcporter: { enabled: true, serverName: "qmd", startDaemon: true },
},
},
} as OpenClawConfig;
spawnMock.mockImplementation((cmd: string, args: string[]) => {
const child = createMockChild({ autoClose: false });
if (cmd === "mcporter" && args[0] === "daemon") {
emitAndClose(child, "stdout", "");
return child;
}
if (cmd === "mcporter" && args[0] === "call") {
emitAndClose(child, "stdout", JSON.stringify({ results: [] }));
return child;
}
emitAndClose(child, "stdout", "[]");
return child;
});
const { manager } = await createManager();
await manager.search("one", { sessionKey: "agent:main:slack:dm:u123" });
await manager.search("two", { sessionKey: "agent:main:slack:dm:u123" });
const daemonStarts = spawnMock.mock.calls.filter(
(call: unknown[]) => call[0] === "mcporter" && (call[1] as string[])[0] === "daemon",
);
expect(daemonStarts).toHaveLength(1);
await manager.close();
});
it("fails closed when no managed collections are configured", async () => {
cfg = {
...cfg,

View File

@@ -25,11 +25,7 @@ import type {
} from "./types.js";
type SqliteDatabase = import("node:sqlite").DatabaseSync;
import type {
ResolvedMemoryBackendConfig,
ResolvedQmdConfig,
ResolvedQmdMcporterConfig,
} from "./backend-config.js";
import type { ResolvedMemoryBackendConfig, ResolvedQmdConfig } from "./backend-config.js";
import { parseQmdQueryJson, type QmdQueryResult } from "./qmd-query-parser.js";
const log = createSubsystemLogger("memory");
@@ -429,37 +425,9 @@ export class QmdMemoryManager implements MemorySearchManager {
return [];
}
const qmdSearchCommand = this.qmd.searchMode;
const mcporterEnabled = this.qmd.mcporter.enabled;
let parsed: QmdQueryResult[];
try {
if (mcporterEnabled) {
const tool: "search" | "vector_search" | "deep_search" =
qmdSearchCommand === "search"
? "search"
: qmdSearchCommand === "vsearch"
? "vector_search"
: "deep_search";
const minScore = opts?.minScore ?? 0;
if (collectionNames.length > 1) {
parsed = await this.runMcporterAcrossCollections({
tool,
query: trimmed,
limit,
minScore,
collectionNames,
});
} else {
parsed = await this.runQmdSearchViaMcporter({
mcporter: this.qmd.mcporter,
tool,
query: trimmed,
limit,
minScore,
collection: collectionNames[0],
timeoutMs: this.qmd.limits.timeoutMs,
});
}
} else if (collectionNames.length > 1) {
if (collectionNames.length > 1) {
parsed = await this.runQueryAcrossCollections(
trimmed,
limit,
@@ -475,11 +443,7 @@ export class QmdMemoryManager implements MemorySearchManager {
parsed = parseQmdQueryJson(result.stdout, result.stderr);
}
} catch (err) {
if (
!mcporterEnabled &&
qmdSearchCommand !== "query" &&
this.isUnsupportedQmdOptionError(err)
) {
if (qmdSearchCommand !== "query" && this.isUnsupportedQmdOptionError(err)) {
log.warn(
`qmd ${qmdSearchCommand} does not support configured flags; retrying search with qmd query`,
);
@@ -499,8 +463,7 @@ export class QmdMemoryManager implements MemorySearchManager {
throw fallbackErr instanceof Error ? fallbackErr : new Error(String(fallbackErr));
}
} else {
const label = mcporterEnabled ? "mcporter/qmd" : `qmd ${qmdSearchCommand}`;
log.warn(`${label} failed: ${String(err)}`);
log.warn(`qmd ${qmdSearchCommand} failed: ${String(err)}`);
throw err instanceof Error ? err : new Error(String(err));
}
}
@@ -896,169 +859,6 @@ export class QmdMemoryManager implements MemorySearchManager {
});
}
private async ensureMcporterDaemonStarted(mcporter: ResolvedQmdMcporterConfig): Promise<void> {
if (!mcporter.enabled) {
return;
}
if (!mcporter.startDaemon) {
type McporterWarnGlobal = typeof globalThis & {
__openclawMcporterColdStartWarned?: boolean;
};
const g: McporterWarnGlobal = globalThis;
if (!g.__openclawMcporterColdStartWarned) {
g.__openclawMcporterColdStartWarned = true;
log.warn(
"mcporter qmd bridge enabled but startDaemon=false; each query may cold-start QMD MCP. Consider setting memory.qmd.mcporter.startDaemon=true to keep it warm.",
);
}
return;
}
type McporterGlobal = typeof globalThis & {
__openclawMcporterDaemonStart?: Promise<void>;
};
const g: McporterGlobal = globalThis;
if (!g.__openclawMcporterDaemonStart) {
g.__openclawMcporterDaemonStart = (async () => {
try {
await this.runMcporter(["daemon", "start"], { timeoutMs: 10_000 });
} catch (err) {
log.warn(`mcporter daemon start failed: ${String(err)}`);
// Allow future searches to retry daemon start on transient failures.
delete g.__openclawMcporterDaemonStart;
}
})();
}
await g.__openclawMcporterDaemonStart;
}
private async runMcporter(
args: string[],
opts?: { timeoutMs?: number },
): Promise<{ stdout: string; stderr: string }> {
return await new Promise((resolve, reject) => {
const child = spawn("mcporter", args, {
// Keep mcporter and direct qmd commands on the same agent-scoped XDG state.
env: this.env,
cwd: this.workspaceDir,
});
let stdout = "";
let stderr = "";
let stdoutTruncated = false;
let stderrTruncated = false;
const timer = opts?.timeoutMs
? setTimeout(() => {
child.kill("SIGKILL");
reject(new Error(`mcporter ${args.join(" ")} timed out after ${opts.timeoutMs}ms`));
}, opts.timeoutMs)
: null;
child.stdout.on("data", (data) => {
const next = appendOutputWithCap(stdout, data.toString("utf8"), this.maxQmdOutputChars);
stdout = next.text;
stdoutTruncated = stdoutTruncated || next.truncated;
});
child.stderr.on("data", (data) => {
const next = appendOutputWithCap(stderr, data.toString("utf8"), this.maxQmdOutputChars);
stderr = next.text;
stderrTruncated = stderrTruncated || next.truncated;
});
child.on("error", (err) => {
if (timer) {
clearTimeout(timer);
}
reject(err);
});
child.on("close", (code) => {
if (timer) {
clearTimeout(timer);
}
if (stdoutTruncated || stderrTruncated) {
reject(
new Error(
`mcporter ${args.join(" ")} produced too much output (limit ${this.maxQmdOutputChars} chars)`,
),
);
return;
}
if (code === 0) {
resolve({ stdout, stderr });
} else {
reject(
new Error(`mcporter ${args.join(" ")} failed (code ${code}): ${stderr || stdout}`),
);
}
});
});
}
private async runQmdSearchViaMcporter(params: {
mcporter: ResolvedQmdMcporterConfig;
tool: "search" | "vector_search" | "deep_search";
query: string;
limit: number;
minScore: number;
collection?: string;
timeoutMs: number;
}): Promise<QmdQueryResult[]> {
await this.ensureMcporterDaemonStarted(params.mcporter);
const selector = `${params.mcporter.serverName}.${params.tool}`;
const callArgs: Record<string, unknown> = {
query: params.query,
limit: params.limit,
minScore: params.minScore,
};
if (params.collection) {
callArgs.collection = params.collection;
}
const result = await this.runMcporter(
[
"call",
selector,
"--args",
JSON.stringify(callArgs),
"--output",
"json",
"--timeout",
String(Math.max(0, params.timeoutMs)),
],
{ timeoutMs: Math.max(params.timeoutMs + 2_000, 5_000) },
);
const parsedUnknown: unknown = JSON.parse(result.stdout);
const isRecord = (value: unknown): value is Record<string, unknown> =>
typeof value === "object" && value !== null && !Array.isArray(value);
const structured =
isRecord(parsedUnknown) && isRecord(parsedUnknown.structuredContent)
? parsedUnknown.structuredContent
: parsedUnknown;
const results: unknown[] =
isRecord(structured) && Array.isArray(structured.results)
? (structured.results as unknown[])
: Array.isArray(structured)
? structured
: [];
const out: QmdQueryResult[] = [];
for (const item of results) {
if (!isRecord(item)) {
continue;
}
const docidRaw = item.docid;
const docid = typeof docidRaw === "string" ? docidRaw.replace(/^#/, "").trim() : "";
if (!docid) {
continue;
}
const scoreRaw = item.score;
const score = typeof scoreRaw === "number" ? scoreRaw : Number(scoreRaw);
const snippet = typeof item.snippet === "string" ? item.snippet : "";
out.push({ docid, score: Number.isFinite(score) ? score : 0, snippet });
}
return out;
}
private async readPartialText(
absPath: string,
from?: number,
@@ -1607,39 +1407,6 @@ export class QmdMemoryManager implements MemorySearchManager {
return [...bestByDocId.values()].toSorted((a, b) => (b.score ?? 0) - (a.score ?? 0));
}
private async runMcporterAcrossCollections(params: {
tool: "search" | "vector_search" | "deep_search";
query: string;
limit: number;
minScore: number;
collectionNames: string[];
}): Promise<QmdQueryResult[]> {
const bestByDocId = new Map<string, QmdQueryResult>();
for (const collectionName of params.collectionNames) {
const parsed = await this.runQmdSearchViaMcporter({
mcporter: this.qmd.mcporter,
tool: params.tool,
query: params.query,
limit: params.limit,
minScore: params.minScore,
collection: collectionName,
timeoutMs: this.qmd.limits.timeoutMs,
});
for (const entry of parsed) {
if (typeof entry.docid !== "string" || !entry.docid.trim()) {
continue;
}
const prev = bestByDocId.get(entry.docid);
const prevScore = typeof prev?.score === "number" ? prev.score : Number.NEGATIVE_INFINITY;
const nextScore = typeof entry.score === "number" ? entry.score : Number.NEGATIVE_INFINITY;
if (!prev || nextScore > prevScore) {
bestByDocId.set(entry.docid, entry);
}
}
}
return [...bestByDocId.values()].toSorted((a, b) => (b.score ?? 0) - (a.score ?? 0));
}
private listManagedCollectionNames(): string[] {
const seen = new Set<string>();
const names: string[] = [];

View File

@@ -38,63 +38,6 @@ describe("extractKeywords", () => {
expect(keywords).toContain("bug");
});
it("extracts keywords from Korean conversational query", () => {
const keywords = extractKeywords("어제 논의한 배포 전략");
expect(keywords).toContain("논의한");
expect(keywords).toContain("배포");
expect(keywords).toContain("전략");
// Should not include stop words
expect(keywords).not.toContain("어제");
});
it("strips Korean particles to extract stems", () => {
const keywords = extractKeywords("서버에서 발생한 에러를 확인");
expect(keywords).toContain("서버");
expect(keywords).toContain("에러");
expect(keywords).toContain("확인");
});
it("filters Korean stop words including inflected forms", () => {
const keywords = extractKeywords("나는 그리고 그래서");
expect(keywords).not.toContain("나");
expect(keywords).not.toContain("나는");
expect(keywords).not.toContain("그리고");
expect(keywords).not.toContain("그래서");
});
it("filters inflected Korean stop words not explicitly listed", () => {
const keywords = extractKeywords("그녀는 우리는");
expect(keywords).not.toContain("그녀는");
expect(keywords).not.toContain("우리는");
expect(keywords).not.toContain("그녀");
expect(keywords).not.toContain("우리");
});
it("does not produce bogus single-char stems from particle stripping", () => {
const keywords = extractKeywords("논의");
expect(keywords).toContain("논의");
expect(keywords).not.toContain("논");
});
it("strips longest Korean trailing particles first", () => {
const keywords = extractKeywords("기능으로 설명");
expect(keywords).toContain("기능");
expect(keywords).not.toContain("기능으");
});
it("keeps stripped ASCII stems for mixed Korean tokens", () => {
const keywords = extractKeywords("API를 배포했다");
expect(keywords).toContain("api");
expect(keywords).toContain("배포했다");
});
it("handles mixed Korean and English query", () => {
const keywords = extractKeywords("API 배포에 대한 논의");
expect(keywords).toContain("api");
expect(keywords).toContain("배포");
expect(keywords).toContain("논의");
});
it("handles empty query", () => {
expect(extractKeywords("")).toEqual([]);
expect(extractKeywords(" ")).toEqual([]);

View File

@@ -118,161 +118,6 @@ const STOP_WORDS_EN = new Set([
"give",
]);
const STOP_WORDS_KO = new Set([
// Particles (조사)
"은",
"는",
"이",
"가",
"을",
"를",
"의",
"에",
"에서",
"로",
"으로",
"와",
"과",
"도",
"만",
"까지",
"부터",
"한테",
"에게",
"께",
"처럼",
"같이",
"보다",
"마다",
"밖에",
"대로",
// Pronouns (대명사)
"나",
"나는",
"내가",
"나를",
"너",
"우리",
"저",
"저희",
"그",
"그녀",
"그들",
"이것",
"저것",
"그것",
"여기",
"저기",
"거기",
// Common verbs / auxiliaries (일반 동사/보조 동사)
"있다",
"없다",
"하다",
"되다",
"이다",
"아니다",
"보다",
"주다",
"오다",
"가다",
// Nouns (의존 명사 / vague)
"것",
"거",
"등",
"수",
"때",
"곳",
"중",
"분",
// Adverbs
"잘",
"더",
"또",
"매우",
"정말",
"아주",
"많이",
"너무",
"좀",
// Conjunctions
"그리고",
"하지만",
"그래서",
"그런데",
"그러나",
"또는",
"그러면",
// Question words
"왜",
"어떻게",
"뭐",
"언제",
"어디",
"누구",
"무엇",
"어떤",
// Time (vague)
"어제",
"오늘",
"내일",
"최근",
"지금",
"아까",
"나중",
"전에",
// Request words
"제발",
"부탁",
]);
// Common Korean trailing particles to strip from words for tokenization
// Sorted by descending length so longest-match-first is guaranteed.
const KO_TRAILING_PARTICLES = [
"에서",
"으로",
"에게",
"한테",
"처럼",
"같이",
"보다",
"까지",
"부터",
"마다",
"밖에",
"대로",
"은",
"는",
"이",
"가",
"을",
"를",
"의",
"에",
"로",
"와",
"과",
"도",
"만",
].toSorted((a, b) => b.length - a.length);
function stripKoreanTrailingParticle(token: string): string | null {
for (const particle of KO_TRAILING_PARTICLES) {
if (token.length > particle.length && token.endsWith(particle)) {
return token.slice(0, -particle.length);
}
}
return null;
}
function isUsefulKoreanStem(stem: string): boolean {
// Prevent bogus one-syllable stems from words like "논의" -> "논".
if (/[\uac00-\ud7af]/.test(stem)) {
return stem.length >= 2;
}
// Keep stripped ASCII stems for mixed tokens like "API를" -> "api".
return /^[a-z0-9_]+$/i.test(stem);
}
const STOP_WORDS_ZH = new Set([
// Pronouns
"我",
@@ -395,7 +240,7 @@ function isValidKeyword(token: string): boolean {
}
/**
* Simple tokenizer that handles English, Chinese, and Korean text.
* Simple tokenizer that handles both English and Chinese text.
* For Chinese, we do character-based splitting since we don't have a proper segmenter.
* For English, we split on whitespace and punctuation.
*/
@@ -407,7 +252,7 @@ function tokenize(text: string): string[] {
const segments = normalized.split(/[\s\p{P}]+/u).filter(Boolean);
for (const segment of segments) {
// Check if segment contains CJK characters (Chinese)
// Check if segment contains CJK characters
if (/[\u4e00-\u9fff]/.test(segment)) {
// For Chinese, extract character n-grams (unigrams and bigrams)
const chars = Array.from(segment).filter((c) => /[\u4e00-\u9fff]/.test(c));
@@ -417,18 +262,6 @@ function tokenize(text: string): string[] {
for (let i = 0; i < chars.length - 1; i++) {
tokens.push(chars[i] + chars[i + 1]);
}
} else if (/[\uac00-\ud7af\u3131-\u3163]/.test(segment)) {
// For Korean (Hangul syllables and jamo), keep the word as-is unless it is
// effectively a stop word once trailing particles are removed.
const stem = stripKoreanTrailingParticle(segment);
const stemIsStopWord = stem !== null && STOP_WORDS_KO.has(stem);
if (!STOP_WORDS_KO.has(segment) && !stemIsStopWord) {
tokens.push(segment);
}
// Also emit particle-stripped stems when they are useful keywords.
if (stem && !STOP_WORDS_KO.has(stem) && isUsefulKoreanStem(stem)) {
tokens.push(stem);
}
} else {
// For non-CJK, keep as single token
tokens.push(segment);
@@ -453,7 +286,7 @@ export function extractKeywords(query: string): string[] {
for (const token of tokens) {
// Skip stop words
if (STOP_WORDS_EN.has(token) || STOP_WORDS_ZH.has(token) || STOP_WORDS_KO.has(token)) {
if (STOP_WORDS_EN.has(token) || STOP_WORDS_ZH.has(token)) {
continue;
}
// Skip invalid keywords

View File

@@ -211,7 +211,6 @@ export {
clearHistoryEntries,
clearHistoryEntriesIfEnabled,
DEFAULT_GROUP_HISTORY_LIMIT,
evictOldHistoryKeys,
recordPendingHistoryEntry,
recordPendingHistoryEntryIfEnabled,
} from "../auto-reply/reply/history.js";

View File

@@ -1,34 +0,0 @@
import { describe, expect, it } from "vitest";
import type { OpenClawConfig } from "../config/config.js";
import { enablePluginInConfig } from "./enable.js";
describe("enablePluginInConfig", () => {
it("enables a plugin entry", () => {
const cfg: OpenClawConfig = {};
const result = enablePluginInConfig(cfg, "google-antigravity-auth");
expect(result.enabled).toBe(true);
expect(result.config.plugins?.entries?.["google-antigravity-auth"]?.enabled).toBe(true);
});
it("adds plugin to allowlist when allowlist is configured", () => {
const cfg: OpenClawConfig = {
plugins: {
allow: ["memory-core"],
},
};
const result = enablePluginInConfig(cfg, "google-antigravity-auth");
expect(result.enabled).toBe(true);
expect(result.config.plugins?.allow).toEqual(["memory-core", "google-antigravity-auth"]);
});
it("refuses enable when plugin is denylisted", () => {
const cfg: OpenClawConfig = {
plugins: {
deny: ["google-antigravity-auth"],
},
};
const result = enablePluginInConfig(cfg, "google-antigravity-auth");
expect(result.enabled).toBe(false);
expect(result.reason).toBe("blocked by denylist");
});
});

View File

@@ -0,0 +1,94 @@
function failOrUndefined(params: { onMissing: "throw" | "undefined"; message: string }): undefined {
if (params.onMissing === "throw") {
throw new Error(params.message);
}
return undefined;
}
export function decodeJsonPointerToken(token: string): string {
return token.replace(/~1/g, "/").replace(/~0/g, "~");
}
export function encodeJsonPointerToken(token: string): string {
return token.replace(/~/g, "~0").replace(/\//g, "~1");
}
export function readJsonPointer(
root: unknown,
pointer: string,
options: { onMissing?: "throw" | "undefined" } = {},
): unknown {
const onMissing = options.onMissing ?? "throw";
if (!pointer.startsWith("/")) {
return failOrUndefined({
onMissing,
message:
'File-backed secret ids must be absolute JSON pointers (for example: "/providers/openai/apiKey").',
});
}
const tokens = pointer
.slice(1)
.split("/")
.map((token) => decodeJsonPointerToken(token));
let current: unknown = root;
for (const token of tokens) {
if (Array.isArray(current)) {
const index = Number.parseInt(token, 10);
if (!Number.isFinite(index) || index < 0 || index >= current.length) {
return failOrUndefined({
onMissing,
message: `JSON pointer segment "${token}" is out of bounds.`,
});
}
current = current[index];
continue;
}
if (typeof current !== "object" || current === null || Array.isArray(current)) {
return failOrUndefined({
onMissing,
message: `JSON pointer segment "${token}" does not exist.`,
});
}
const record = current as Record<string, unknown>;
if (!Object.hasOwn(record, token)) {
return failOrUndefined({
onMissing,
message: `JSON pointer segment "${token}" does not exist.`,
});
}
current = record[token];
}
return current;
}
export function setJsonPointer(
root: Record<string, unknown>,
pointer: string,
value: unknown,
): void {
if (!pointer.startsWith("/")) {
throw new Error(`Invalid JSON pointer "${pointer}".`);
}
const tokens = pointer
.slice(1)
.split("/")
.map((token) => decodeJsonPointerToken(token));
let current: Record<string, unknown> = root;
for (let index = 0; index < tokens.length; index += 1) {
const token = tokens[index];
const isLast = index === tokens.length - 1;
if (isLast) {
current[token] = value;
return;
}
const child = current[token];
if (typeof child !== "object" || child === null || Array.isArray(child)) {
current[token] = {};
}
current = current[token] as Record<string, unknown>;
}
}

204
src/secrets/migrate.test.ts Normal file
View File

@@ -0,0 +1,204 @@
import fs from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
const runExecMock = vi.hoisted(() => vi.fn());
vi.mock("../process/exec.js", () => ({
runExec: runExecMock,
}));
const { rollbackSecretsMigration, runSecretsMigration } = await import("./migrate.js");
describe("secrets migrate", () => {
let baseDir = "";
let stateDir = "";
let configPath = "";
let env: NodeJS.ProcessEnv;
let authStorePath = "";
let envPath = "";
beforeEach(async () => {
baseDir = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-secrets-migrate-"));
stateDir = path.join(baseDir, ".openclaw");
configPath = path.join(stateDir, "openclaw.json");
authStorePath = path.join(stateDir, "agents", "main", "agent", "auth-profiles.json");
envPath = path.join(stateDir, ".env");
env = {
...process.env,
OPENCLAW_STATE_DIR: stateDir,
OPENCLAW_CONFIG_PATH: configPath,
};
await fs.mkdir(path.dirname(configPath), { recursive: true });
await fs.mkdir(path.dirname(authStorePath), { recursive: true });
await fs.writeFile(
configPath,
`${JSON.stringify(
{
models: {
providers: {
openai: {
baseUrl: "https://api.openai.com/v1",
apiKey: "sk-openai-plaintext",
models: [{ id: "gpt-5", name: "gpt-5" }],
},
},
},
skills: {
entries: {
"review-pr": {
enabled: true,
apiKey: "sk-skill-plaintext",
},
},
},
channels: {
googlechat: {
serviceAccount: '{"type":"service_account","client_email":"bot@example.com"}',
},
},
},
null,
2,
)}\n`,
"utf8",
);
await fs.writeFile(
authStorePath,
`${JSON.stringify(
{
version: 1,
profiles: {
"openai:default": {
type: "api_key",
provider: "openai",
key: "sk-profile-plaintext",
},
},
},
null,
2,
)}\n`,
"utf8",
);
await fs.writeFile(
envPath,
"OPENAI_API_KEY=sk-openai-plaintext\nSKILL_KEY=sk-skill-plaintext\nUNRELATED=value\n",
"utf8",
);
runExecMock.mockReset();
runExecMock.mockImplementation(async (_cmd: string, args: string[]) => {
if (args[0] === "--encrypt") {
const outputPath = args[args.indexOf("--output") + 1];
const inputPath = args.at(-1);
if (!outputPath || !inputPath) {
throw new Error("missing sops encrypt paths");
}
await fs.copyFile(inputPath, outputPath);
return { stdout: "", stderr: "" };
}
if (args[0] === "--decrypt") {
const sourcePath = args.at(-1);
if (!sourcePath) {
throw new Error("missing sops decrypt source");
}
const raw = await fs.readFile(sourcePath, "utf8");
return { stdout: raw, stderr: "" };
}
throw new Error(`unexpected sops invocation: ${args.join(" ")}`);
});
});
afterEach(async () => {
await fs.rm(baseDir, { recursive: true, force: true });
});
it("reports a dry-run without mutating files", async () => {
const beforeConfig = await fs.readFile(configPath, "utf8");
const beforeAuthStore = await fs.readFile(authStorePath, "utf8");
const result = await runSecretsMigration({ env });
expect(result.mode).toBe("dry-run");
expect(result.changed).toBe(true);
expect(result.counters.secretsWritten).toBeGreaterThanOrEqual(3);
expect(await fs.readFile(configPath, "utf8")).toBe(beforeConfig);
expect(await fs.readFile(authStorePath, "utf8")).toBe(beforeAuthStore);
});
it("migrates plaintext to file-backed refs and can rollback", async () => {
const applyResult = await runSecretsMigration({ env, write: true });
expect(applyResult.mode).toBe("write");
expect(applyResult.changed).toBe(true);
expect(applyResult.backupId).toBeTruthy();
const migratedConfig = JSON.parse(await fs.readFile(configPath, "utf8")) as {
models: { providers: { openai: { apiKey: unknown } } };
skills: { entries: { "review-pr": { apiKey: unknown } } };
channels: { googlechat: { serviceAccount?: unknown; serviceAccountRef?: unknown } };
secrets: { sources: { file: { type: string; path: string } } };
};
expect(migratedConfig.models.providers.openai.apiKey).toEqual({
source: "file",
id: "/providers/openai/apiKey",
});
expect(migratedConfig.skills.entries["review-pr"].apiKey).toEqual({
source: "file",
id: "/skills/entries/review-pr/apiKey",
});
expect(migratedConfig.channels.googlechat.serviceAccount).toBeUndefined();
expect(migratedConfig.channels.googlechat.serviceAccountRef).toEqual({
source: "file",
id: "/channels/googlechat/serviceAccount",
});
expect(migratedConfig.secrets.sources.file.type).toBe("sops");
const migratedAuth = JSON.parse(await fs.readFile(authStorePath, "utf8")) as {
profiles: { "openai:default": { key?: string; keyRef?: unknown } };
};
expect(migratedAuth.profiles["openai:default"].key).toBeUndefined();
expect(migratedAuth.profiles["openai:default"].keyRef).toEqual({
source: "file",
id: "/auth-profiles/main/openai:default/key",
});
const migratedEnv = await fs.readFile(envPath, "utf8");
expect(migratedEnv).not.toContain("sk-openai-plaintext");
expect(migratedEnv).toContain("SKILL_KEY=sk-skill-plaintext");
expect(migratedEnv).toContain("UNRELATED=value");
const secretsPath = path.join(stateDir, "secrets.enc.json");
const secretsPayload = JSON.parse(await fs.readFile(secretsPath, "utf8")) as {
providers: { openai: { apiKey: string } };
skills: { entries: { "review-pr": { apiKey: string } } };
channels: { googlechat: { serviceAccount: string } };
"auth-profiles": { main: { "openai:default": { key: string } } };
};
expect(secretsPayload.providers.openai.apiKey).toBe("sk-openai-plaintext");
expect(secretsPayload.skills.entries["review-pr"].apiKey).toBe("sk-skill-plaintext");
expect(secretsPayload.channels.googlechat.serviceAccount).toContain("service_account");
expect(secretsPayload["auth-profiles"].main["openai:default"].key).toBe("sk-profile-plaintext");
const rollbackResult = await rollbackSecretsMigration({ env, backupId: applyResult.backupId! });
expect(rollbackResult.restoredFiles).toBeGreaterThan(0);
const rolledBackConfig = await fs.readFile(configPath, "utf8");
expect(rolledBackConfig).toContain("sk-openai-plaintext");
expect(rolledBackConfig).toContain("sk-skill-plaintext");
const rolledBackAuth = await fs.readFile(authStorePath, "utf8");
expect(rolledBackAuth).toContain("sk-profile-plaintext");
await expect(fs.stat(secretsPath)).rejects.toThrow();
const rolledBackEnv = await fs.readFile(envPath, "utf8");
expect(rolledBackEnv).toContain("OPENAI_API_KEY=sk-openai-plaintext");
});
});

886
src/secrets/migrate.ts Normal file
View File

@@ -0,0 +1,886 @@
import crypto from "node:crypto";
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
import { isDeepStrictEqual } from "node:util";
import { listAgentIds, resolveAgentDir } from "../agents/agent-scope.js";
import { resolveAuthStorePath } from "../agents/auth-profiles/paths.js";
import { createConfigIO, resolveStateDir, type OpenClawConfig } from "../config/config.js";
import { isSecretRef } from "../config/types.secrets.js";
import { resolveConfigDir, resolveUserPath } from "../utils.js";
import {
encodeJsonPointerToken,
readJsonPointer as readJsonPointerRaw,
setJsonPointer,
} from "./json-pointer.js";
import { listKnownSecretEnvVarNames } from "./provider-env-vars.js";
import { decryptSopsJsonFile, encryptSopsJsonFile, DEFAULT_SOPS_TIMEOUT_MS } from "./sops.js";
const DEFAULT_SECRETS_FILE_PATH = "~/.openclaw/secrets.enc.json";
const BACKUP_DIRNAME = "secrets-migrate";
const BACKUP_MANIFEST_FILENAME = "manifest.json";
const BACKUP_RETENTION = 20;
type MigrationCounters = {
configRefs: number;
authProfileRefs: number;
plaintextRemoved: number;
secretsWritten: number;
envEntriesRemoved: number;
authStoresChanged: number;
};
type AuthStoreChange = {
path: string;
nextStore: Record<string, unknown>;
};
type EnvChange = {
path: string;
nextRaw: string;
};
type BackupManifestEntry = {
path: string;
existed: boolean;
backupPath?: string;
mode?: number;
};
type BackupManifest = {
version: 1;
backupId: string;
createdAt: string;
entries: BackupManifestEntry[];
};
type MigrationPlan = {
changed: boolean;
counters: MigrationCounters;
stateDir: string;
configChanged: boolean;
nextConfig: OpenClawConfig;
configWriteOptions: Awaited<
ReturnType<ReturnType<typeof createConfigIO>["readConfigFileSnapshotForWrite"]>
>["writeOptions"];
authStoreChanges: AuthStoreChange[];
payloadChanged: boolean;
nextPayload: Record<string, unknown>;
secretsFilePath: string;
secretsFileTimeoutMs: number;
envChange: EnvChange | null;
backupTargets: string[];
};
export type SecretsMigrationRunOptions = {
write?: boolean;
scrubEnv?: boolean;
env?: NodeJS.ProcessEnv;
now?: Date;
};
export type SecretsMigrationRunResult = {
mode: "dry-run" | "write";
changed: boolean;
backupId?: string;
backupDir?: string;
secretsFilePath: string;
counters: MigrationCounters;
changedFiles: string[];
};
export type SecretsMigrationRollbackOptions = {
backupId: string;
env?: NodeJS.ProcessEnv;
};
export type SecretsMigrationRollbackResult = {
backupId: string;
restoredFiles: number;
deletedFiles: number;
};
function isRecord(value: unknown): value is Record<string, unknown> {
return typeof value === "object" && value !== null && !Array.isArray(value);
}
function isNonEmptyString(value: unknown): value is string {
return typeof value === "string" && value.trim().length > 0;
}
function normalizeSopsTimeoutMs(value: unknown): number {
if (typeof value === "number" && Number.isFinite(value)) {
return Math.max(1, Math.floor(value));
}
return DEFAULT_SOPS_TIMEOUT_MS;
}
function readJsonPointer(root: unknown, pointer: string): unknown {
return readJsonPointerRaw(root, pointer, { onMissing: "undefined" });
}
function formatBackupId(now: Date): string {
const year = now.getUTCFullYear();
const month = String(now.getUTCMonth() + 1).padStart(2, "0");
const day = String(now.getUTCDate()).padStart(2, "0");
const hour = String(now.getUTCHours()).padStart(2, "0");
const minute = String(now.getUTCMinutes()).padStart(2, "0");
const second = String(now.getUTCSeconds()).padStart(2, "0");
return `${year}${month}${day}T${hour}${minute}${second}Z`;
}
function parseEnvValue(raw: string): string {
const trimmed = raw.trim();
if (
(trimmed.startsWith('"') && trimmed.endsWith('"')) ||
(trimmed.startsWith("'") && trimmed.endsWith("'"))
) {
return trimmed.slice(1, -1);
}
return trimmed;
}
function scrubEnvRaw(
raw: string,
migratedValues: Set<string>,
allowedEnvKeys: Set<string>,
): {
nextRaw: string;
removed: number;
} {
if (migratedValues.size === 0 || allowedEnvKeys.size === 0) {
return { nextRaw: raw, removed: 0 };
}
const lines = raw.split(/\r?\n/);
const nextLines: string[] = [];
let removed = 0;
for (const line of lines) {
const match = line.match(/^\s*(?:export\s+)?([A-Za-z_][A-Za-z0-9_]*)\s*=\s*(.*)$/);
if (!match) {
nextLines.push(line);
continue;
}
const envKey = match[1] ?? "";
if (!allowedEnvKeys.has(envKey)) {
nextLines.push(line);
continue;
}
const parsedValue = parseEnvValue(match[2] ?? "");
if (migratedValues.has(parsedValue)) {
removed += 1;
continue;
}
nextLines.push(line);
}
const hadTrailingNewline = raw.endsWith("\n");
const joined = nextLines.join("\n");
return {
nextRaw:
hadTrailingNewline || joined.length === 0
? `${joined}${joined.endsWith("\n") ? "" : "\n"}`
: joined,
removed,
};
}
function ensureDirForFile(filePath: string): void {
fs.mkdirSync(path.dirname(filePath), { recursive: true, mode: 0o700 });
}
function saveJsonFile(pathname: string, value: unknown): void {
ensureDirForFile(pathname);
fs.writeFileSync(pathname, `${JSON.stringify(value, null, 2)}\n`, "utf8");
fs.chmodSync(pathname, 0o600);
}
function resolveFileSource(
config: OpenClawConfig,
env: NodeJS.ProcessEnv,
): {
path: string;
timeoutMs: number;
hadConfiguredSource: boolean;
} {
const source = config.secrets?.sources?.file;
if (source && source.type === "sops" && isNonEmptyString(source.path)) {
return {
path: resolveUserPath(source.path),
timeoutMs: normalizeSopsTimeoutMs(source.timeoutMs),
hadConfiguredSource: true,
};
}
return {
path: resolveUserPath(resolveDefaultSecretsConfigPath(env)),
timeoutMs: DEFAULT_SOPS_TIMEOUT_MS,
hadConfiguredSource: false,
};
}
function resolveDefaultSecretsConfigPath(env: NodeJS.ProcessEnv): string {
if (env.OPENCLAW_STATE_DIR?.trim() || env.CLAWDBOT_STATE_DIR?.trim()) {
return path.join(resolveStateDir(env, os.homedir), "secrets.enc.json");
}
return DEFAULT_SECRETS_FILE_PATH;
}
async function decryptSopsJson(
pathname: string,
timeoutMs: number,
): Promise<Record<string, unknown>> {
if (!fs.existsSync(pathname)) {
return {};
}
const parsed = await decryptSopsJsonFile({
path: pathname,
timeoutMs,
missingBinaryMessage:
"sops binary not found in PATH. Install sops >= 3.9.0 to run secrets migrate.",
});
if (!isRecord(parsed)) {
throw new Error("sops decrypt failed: decrypted payload is not a JSON object");
}
return parsed;
}
async function encryptSopsJson(params: {
pathname: string;
timeoutMs: number;
payload: Record<string, unknown>;
}): Promise<void> {
await encryptSopsJsonFile({
path: params.pathname,
payload: params.payload,
timeoutMs: params.timeoutMs,
missingBinaryMessage:
"sops binary not found in PATH. Install sops >= 3.9.0 to run secrets migrate.",
});
}
function migrateModelProviderSecrets(params: {
config: OpenClawConfig;
payload: Record<string, unknown>;
counters: MigrationCounters;
migratedValues: Set<string>;
}): void {
const providers = params.config.models?.providers as
| Record<string, { apiKey?: unknown }>
| undefined;
if (!providers) {
return;
}
for (const [providerId, provider] of Object.entries(providers)) {
if (isSecretRef(provider.apiKey)) {
continue;
}
if (!isNonEmptyString(provider.apiKey)) {
continue;
}
const value = provider.apiKey.trim();
const id = `/providers/${encodeJsonPointerToken(providerId)}/apiKey`;
const existing = readJsonPointer(params.payload, id);
if (!isDeepStrictEqual(existing, value)) {
setJsonPointer(params.payload, id, value);
params.counters.secretsWritten += 1;
}
provider.apiKey = { source: "file", id };
params.counters.configRefs += 1;
params.migratedValues.add(value);
}
}
function migrateSkillEntrySecrets(params: {
config: OpenClawConfig;
payload: Record<string, unknown>;
counters: MigrationCounters;
migratedValues: Set<string>;
}): void {
const entries = params.config.skills?.entries as Record<string, { apiKey?: unknown }> | undefined;
if (!entries) {
return;
}
for (const [skillKey, entry] of Object.entries(entries)) {
if (!isRecord(entry) || isSecretRef(entry.apiKey)) {
continue;
}
if (!isNonEmptyString(entry.apiKey)) {
continue;
}
const value = entry.apiKey.trim();
const id = `/skills/entries/${encodeJsonPointerToken(skillKey)}/apiKey`;
const existing = readJsonPointer(params.payload, id);
if (!isDeepStrictEqual(existing, value)) {
setJsonPointer(params.payload, id, value);
params.counters.secretsWritten += 1;
}
entry.apiKey = { source: "file", id };
params.counters.configRefs += 1;
params.migratedValues.add(value);
}
}
function migrateGoogleChatServiceAccount(params: {
account: Record<string, unknown>;
pointerId: string;
counters: MigrationCounters;
payload: Record<string, unknown>;
}): void {
const explicitRef = isSecretRef(params.account.serviceAccountRef)
? params.account.serviceAccountRef
: null;
const inlineRef = isSecretRef(params.account.serviceAccount)
? params.account.serviceAccount
: null;
if (explicitRef || inlineRef) {
if (
params.account.serviceAccount !== undefined &&
!isSecretRef(params.account.serviceAccount)
) {
delete params.account.serviceAccount;
params.counters.plaintextRemoved += 1;
}
return;
}
const value = params.account.serviceAccount;
const hasStringValue = isNonEmptyString(value);
const hasObjectValue = isRecord(value) && Object.keys(value).length > 0;
if (!hasStringValue && !hasObjectValue) {
return;
}
const id = `${params.pointerId}/serviceAccount`;
const normalizedValue = hasStringValue ? value.trim() : structuredClone(value);
const existing = readJsonPointer(params.payload, id);
if (!isDeepStrictEqual(existing, normalizedValue)) {
setJsonPointer(params.payload, id, normalizedValue);
params.counters.secretsWritten += 1;
}
params.account.serviceAccountRef = { source: "file", id };
delete params.account.serviceAccount;
params.counters.configRefs += 1;
}
function migrateGoogleChatSecrets(params: {
config: OpenClawConfig;
payload: Record<string, unknown>;
counters: MigrationCounters;
}): void {
const googlechat = params.config.channels?.googlechat;
if (!isRecord(googlechat)) {
return;
}
migrateGoogleChatServiceAccount({
account: googlechat,
pointerId: "/channels/googlechat",
payload: params.payload,
counters: params.counters,
});
if (!isRecord(googlechat.accounts)) {
return;
}
for (const [accountId, accountValue] of Object.entries(googlechat.accounts)) {
if (!isRecord(accountValue)) {
continue;
}
migrateGoogleChatServiceAccount({
account: accountValue,
pointerId: `/channels/googlechat/accounts/${encodeJsonPointerToken(accountId)}`,
payload: params.payload,
counters: params.counters,
});
}
}
function collectAuthStorePaths(config: OpenClawConfig, stateDir: string): string[] {
const paths = new Set<string>();
paths.add(resolveUserPath(resolveAuthStorePath()));
const agentsRoot = path.join(resolveUserPath(stateDir), "agents");
if (fs.existsSync(agentsRoot)) {
for (const entry of fs.readdirSync(agentsRoot, { withFileTypes: true })) {
if (!entry.isDirectory()) {
continue;
}
paths.add(path.join(agentsRoot, entry.name, "agent", "auth-profiles.json"));
}
}
for (const agentId of listAgentIds(config)) {
const agentDir = resolveAgentDir(config, agentId);
paths.add(resolveUserPath(resolveAuthStorePath(agentDir)));
}
return [...paths];
}
function deriveAuthStoreScope(authStorePath: string, stateDir: string): string {
const agentsRoot = path.join(resolveUserPath(stateDir), "agents");
const relative = path.relative(agentsRoot, authStorePath);
if (!relative.startsWith("..")) {
const segments = relative.split(path.sep);
if (segments.length >= 3 && segments[1] === "agent" && segments[2] === "auth-profiles.json") {
const candidate = segments[0]?.trim();
if (candidate) {
return candidate;
}
}
}
const digest = crypto.createHash("sha1").update(authStorePath).digest("hex").slice(0, 8);
return `path-${digest}`;
}
function migrateAuthStoreSecrets(params: {
store: Record<string, unknown>;
scope: string;
payload: Record<string, unknown>;
counters: MigrationCounters;
migratedValues: Set<string>;
}): boolean {
const profiles = params.store.profiles;
if (!isRecord(profiles)) {
return false;
}
let changed = false;
for (const [profileId, profileValue] of Object.entries(profiles)) {
if (!isRecord(profileValue)) {
continue;
}
if (profileValue.type === "api_key") {
const keyRef = isSecretRef(profileValue.keyRef) ? profileValue.keyRef : null;
const key = isNonEmptyString(profileValue.key) ? profileValue.key.trim() : "";
if (keyRef) {
if (key) {
delete profileValue.key;
params.counters.plaintextRemoved += 1;
changed = true;
}
continue;
}
if (!key) {
continue;
}
const id = `/auth-profiles/${encodeJsonPointerToken(params.scope)}/${encodeJsonPointerToken(profileId)}/key`;
const existing = readJsonPointer(params.payload, id);
if (!isDeepStrictEqual(existing, key)) {
setJsonPointer(params.payload, id, key);
params.counters.secretsWritten += 1;
}
profileValue.keyRef = { source: "file", id };
delete profileValue.key;
params.counters.authProfileRefs += 1;
params.migratedValues.add(key);
changed = true;
continue;
}
if (profileValue.type === "token") {
const tokenRef = isSecretRef(profileValue.tokenRef) ? profileValue.tokenRef : null;
const token = isNonEmptyString(profileValue.token) ? profileValue.token.trim() : "";
if (tokenRef) {
if (token) {
delete profileValue.token;
params.counters.plaintextRemoved += 1;
changed = true;
}
continue;
}
if (!token) {
continue;
}
const id = `/auth-profiles/${encodeJsonPointerToken(params.scope)}/${encodeJsonPointerToken(profileId)}/token`;
const existing = readJsonPointer(params.payload, id);
if (!isDeepStrictEqual(existing, token)) {
setJsonPointer(params.payload, id, token);
params.counters.secretsWritten += 1;
}
profileValue.tokenRef = { source: "file", id };
delete profileValue.token;
params.counters.authProfileRefs += 1;
params.migratedValues.add(token);
changed = true;
}
}
return changed;
}
function resolveBackupRoot(stateDir: string): string {
return path.join(resolveUserPath(stateDir), "backups", BACKUP_DIRNAME);
}
function createBackupManifest(params: {
stateDir: string;
targets: string[];
backupId: string;
now: Date;
}): { backupDir: string; manifestPath: string; manifest: BackupManifest } {
const backupDir = path.join(resolveBackupRoot(params.stateDir), params.backupId);
fs.mkdirSync(backupDir, { recursive: true, mode: 0o700 });
const entries: BackupManifestEntry[] = [];
let index = 0;
for (const target of params.targets) {
const normalized = resolveUserPath(target);
const exists = fs.existsSync(normalized);
if (!exists) {
entries.push({ path: normalized, existed: false });
continue;
}
const backupName = `file-${String(index).padStart(4, "0")}.bak`;
const backupPath = path.join(backupDir, backupName);
fs.copyFileSync(normalized, backupPath);
const stats = fs.statSync(normalized);
entries.push({
path: normalized,
existed: true,
backupPath,
mode: stats.mode & 0o777,
});
index += 1;
}
const manifest: BackupManifest = {
version: 1,
backupId: params.backupId,
createdAt: params.now.toISOString(),
entries,
};
const manifestPath = path.join(backupDir, BACKUP_MANIFEST_FILENAME);
fs.writeFileSync(manifestPath, `${JSON.stringify(manifest, null, 2)}\n`, "utf8");
fs.chmodSync(manifestPath, 0o600);
return { backupDir, manifestPath, manifest };
}
function restoreFromManifest(manifest: BackupManifest): {
restoredFiles: number;
deletedFiles: number;
} {
let restoredFiles = 0;
let deletedFiles = 0;
for (const entry of manifest.entries) {
if (!entry.existed) {
if (fs.existsSync(entry.path)) {
fs.rmSync(entry.path, { force: true });
deletedFiles += 1;
}
continue;
}
if (!entry.backupPath || !fs.existsSync(entry.backupPath)) {
throw new Error(`Backup file is missing for ${entry.path}.`);
}
ensureDirForFile(entry.path);
fs.copyFileSync(entry.backupPath, entry.path);
fs.chmodSync(entry.path, entry.mode ?? 0o600);
restoredFiles += 1;
}
return { restoredFiles, deletedFiles };
}
function pruneOldBackups(stateDir: string): void {
const backupRoot = resolveBackupRoot(stateDir);
if (!fs.existsSync(backupRoot)) {
return;
}
const dirs = fs
.readdirSync(backupRoot, { withFileTypes: true })
.filter((entry) => entry.isDirectory())
.map((entry) => entry.name)
.toSorted();
if (dirs.length <= BACKUP_RETENTION) {
return;
}
const toDelete = dirs.slice(0, Math.max(0, dirs.length - BACKUP_RETENTION));
for (const dir of toDelete) {
fs.rmSync(path.join(backupRoot, dir), { recursive: true, force: true });
}
}
async function buildMigrationPlan(params: {
env: NodeJS.ProcessEnv;
scrubEnv: boolean;
}): Promise<MigrationPlan> {
const io = createConfigIO({ env: params.env });
const { snapshot, writeOptions } = await io.readConfigFileSnapshotForWrite();
if (!snapshot.valid) {
const issues =
snapshot.issues.length > 0
? snapshot.issues.map((issue) => `${issue.path || "<root>"}: ${issue.message}`).join("\n")
: "Unknown validation issue.";
throw new Error(`Cannot migrate secrets because config is invalid:\n${issues}`);
}
const stateDir = resolveStateDir(params.env, os.homedir);
const nextConfig = structuredClone(snapshot.config);
const fileSource = resolveFileSource(nextConfig, params.env);
const previousPayload = await decryptSopsJson(fileSource.path, fileSource.timeoutMs);
const nextPayload = structuredClone(previousPayload);
const counters: MigrationCounters = {
configRefs: 0,
authProfileRefs: 0,
plaintextRemoved: 0,
secretsWritten: 0,
envEntriesRemoved: 0,
authStoresChanged: 0,
};
const migratedValues = new Set<string>();
migrateModelProviderSecrets({
config: nextConfig,
payload: nextPayload,
counters,
migratedValues,
});
migrateSkillEntrySecrets({
config: nextConfig,
payload: nextPayload,
counters,
migratedValues,
});
migrateGoogleChatSecrets({
config: nextConfig,
payload: nextPayload,
counters,
});
const authStoreChanges: AuthStoreChange[] = [];
for (const authStorePath of collectAuthStorePaths(nextConfig, stateDir)) {
if (!fs.existsSync(authStorePath)) {
continue;
}
const raw = fs.readFileSync(authStorePath, "utf8");
let parsed: unknown;
try {
parsed = JSON.parse(raw) as unknown;
} catch {
continue;
}
if (!isRecord(parsed)) {
continue;
}
const nextStore = structuredClone(parsed);
const scope = deriveAuthStoreScope(authStorePath, stateDir);
const changed = migrateAuthStoreSecrets({
store: nextStore,
scope,
payload: nextPayload,
counters,
migratedValues,
});
if (!changed) {
continue;
}
authStoreChanges.push({ path: authStorePath, nextStore });
}
counters.authStoresChanged = authStoreChanges.length;
if (counters.secretsWritten > 0 && !fileSource.hadConfiguredSource) {
const defaultConfigPath = resolveDefaultSecretsConfigPath(params.env);
nextConfig.secrets ??= {};
nextConfig.secrets.sources ??= {};
nextConfig.secrets.sources.file = {
type: "sops",
path: defaultConfigPath,
timeoutMs: DEFAULT_SOPS_TIMEOUT_MS,
};
}
const configChanged = !isDeepStrictEqual(snapshot.config, nextConfig);
const payloadChanged = !isDeepStrictEqual(previousPayload, nextPayload);
let envChange: EnvChange | null = null;
if (params.scrubEnv && migratedValues.size > 0) {
const envPath = path.join(resolveConfigDir(params.env, os.homedir), ".env");
if (fs.existsSync(envPath)) {
const rawEnv = fs.readFileSync(envPath, "utf8");
const scrubbed = scrubEnvRaw(rawEnv, migratedValues, new Set(listKnownSecretEnvVarNames()));
if (scrubbed.removed > 0 && scrubbed.nextRaw !== rawEnv) {
counters.envEntriesRemoved = scrubbed.removed;
envChange = {
path: envPath,
nextRaw: scrubbed.nextRaw,
};
}
}
}
const backupTargets = new Set<string>();
if (configChanged) {
backupTargets.add(io.configPath);
}
if (payloadChanged) {
backupTargets.add(fileSource.path);
}
for (const change of authStoreChanges) {
backupTargets.add(change.path);
}
if (envChange) {
backupTargets.add(envChange.path);
}
return {
changed: configChanged || payloadChanged || authStoreChanges.length > 0 || Boolean(envChange),
counters,
stateDir,
configChanged,
nextConfig,
configWriteOptions: writeOptions,
authStoreChanges,
payloadChanged,
nextPayload,
secretsFilePath: fileSource.path,
secretsFileTimeoutMs: fileSource.timeoutMs,
envChange,
backupTargets: [...backupTargets],
};
}
export async function runSecretsMigration(
options: SecretsMigrationRunOptions = {},
): Promise<SecretsMigrationRunResult> {
const env = options.env ?? process.env;
const scrubEnv = options.scrubEnv ?? true;
const plan = await buildMigrationPlan({ env, scrubEnv });
if (!options.write) {
return {
mode: "dry-run",
changed: plan.changed,
secretsFilePath: plan.secretsFilePath,
counters: plan.counters,
changedFiles: plan.backupTargets,
};
}
if (!plan.changed) {
return {
mode: "write",
changed: false,
secretsFilePath: plan.secretsFilePath,
counters: plan.counters,
changedFiles: [],
};
}
const now = options.now ?? new Date();
const backupId = formatBackupId(now);
const backup = createBackupManifest({
stateDir: plan.stateDir,
targets: plan.backupTargets,
backupId,
now,
});
try {
if (plan.payloadChanged) {
await encryptSopsJson({
pathname: plan.secretsFilePath,
timeoutMs: plan.secretsFileTimeoutMs,
payload: plan.nextPayload,
});
}
if (plan.configChanged) {
const io = createConfigIO({ env });
await io.writeConfigFile(plan.nextConfig, plan.configWriteOptions);
}
for (const change of plan.authStoreChanges) {
saveJsonFile(change.path, change.nextStore);
}
if (plan.envChange) {
ensureDirForFile(plan.envChange.path);
fs.writeFileSync(plan.envChange.path, plan.envChange.nextRaw, "utf8");
fs.chmodSync(plan.envChange.path, 0o600);
}
} catch (err) {
restoreFromManifest(backup.manifest);
throw new Error(
`Secrets migration failed and was rolled back from backup ${backupId}: ${String(err)}`,
{
cause: err,
},
);
}
pruneOldBackups(plan.stateDir);
return {
mode: "write",
changed: true,
backupId,
backupDir: backup.backupDir,
secretsFilePath: plan.secretsFilePath,
counters: plan.counters,
changedFiles: plan.backupTargets,
};
}
export function resolveSecretsMigrationBackupRoot(env: NodeJS.ProcessEnv = process.env): string {
return resolveBackupRoot(resolveStateDir(env, os.homedir));
}
export function listSecretsMigrationBackups(env: NodeJS.ProcessEnv = process.env): string[] {
const root = resolveSecretsMigrationBackupRoot(env);
if (!fs.existsSync(root)) {
return [];
}
return fs
.readdirSync(root, { withFileTypes: true })
.filter((entry) => entry.isDirectory())
.map((entry) => entry.name)
.toSorted();
}
export async function rollbackSecretsMigration(
options: SecretsMigrationRollbackOptions,
): Promise<SecretsMigrationRollbackResult> {
const env = options.env ?? process.env;
const backupDir = path.join(resolveSecretsMigrationBackupRoot(env), options.backupId);
const manifestPath = path.join(backupDir, BACKUP_MANIFEST_FILENAME);
if (!fs.existsSync(manifestPath)) {
const available = listSecretsMigrationBackups(env);
const suffix =
available.length > 0
? ` Available backups: ${available.slice(-10).join(", ")}`
: " No backups were found.";
throw new Error(`Backup "${options.backupId}" was not found.${suffix}`);
}
let parsed: unknown;
try {
parsed = JSON.parse(fs.readFileSync(manifestPath, "utf8")) as unknown;
} catch (err) {
throw new Error(`Failed to read backup manifest at ${manifestPath}: ${String(err)}`, {
cause: err,
});
}
if (!isRecord(parsed) || !Array.isArray(parsed.entries)) {
throw new Error(`Backup manifest at ${manifestPath} is invalid.`);
}
const manifest = parsed as BackupManifest;
const restored = restoreFromManifest(manifest);
return {
backupId: options.backupId,
restoredFiles: restored.restoredFiles,
deletedFiles: restored.deletedFiles,
};
}

View File

@@ -0,0 +1,28 @@
export const PROVIDER_ENV_VARS: Record<string, readonly string[]> = {
openai: ["OPENAI_API_KEY"],
anthropic: ["ANTHROPIC_API_KEY"],
google: ["GEMINI_API_KEY"],
minimax: ["MINIMAX_API_KEY"],
"minimax-cn": ["MINIMAX_API_KEY"],
moonshot: ["MOONSHOT_API_KEY"],
"kimi-coding": ["KIMI_API_KEY", "KIMICODE_API_KEY"],
synthetic: ["SYNTHETIC_API_KEY"],
venice: ["VENICE_API_KEY"],
zai: ["ZAI_API_KEY", "Z_AI_API_KEY"],
xiaomi: ["XIAOMI_API_KEY"],
openrouter: ["OPENROUTER_API_KEY"],
"cloudflare-ai-gateway": ["CLOUDFLARE_AI_GATEWAY_API_KEY"],
litellm: ["LITELLM_API_KEY"],
"vercel-ai-gateway": ["AI_GATEWAY_API_KEY"],
opencode: ["OPENCODE_API_KEY", "OPENCODE_ZEN_API_KEY"],
together: ["TOGETHER_API_KEY"],
huggingface: ["HUGGINGFACE_HUB_TOKEN", "HF_TOKEN"],
qianfan: ["QIANFAN_API_KEY"],
xai: ["XAI_API_KEY"],
volcengine: ["VOLCANO_ENGINE_API_KEY"],
byteplus: ["BYTEPLUS_API_KEY"],
};
export function listKnownSecretEnvVarNames(): string[] {
return [...new Set(Object.values(PROVIDER_ENV_VARS).flatMap((keys) => keys))];
}

169
src/secrets/runtime.test.ts Normal file
View File

@@ -0,0 +1,169 @@
import { afterEach, describe, expect, it, vi } from "vitest";
import { ensureAuthProfileStore } from "../agents/auth-profiles.js";
import { loadConfig, type OpenClawConfig } from "../config/config.js";
import {
activateSecretsRuntimeSnapshot,
clearSecretsRuntimeSnapshot,
prepareSecretsRuntimeSnapshot,
} from "./runtime.js";
const runExecMock = vi.hoisted(() => vi.fn());
vi.mock("../process/exec.js", () => ({
runExec: runExecMock,
}));
describe("secrets runtime snapshot", () => {
afterEach(() => {
runExecMock.mockReset();
clearSecretsRuntimeSnapshot();
});
it("resolves env refs for config and auth profiles", async () => {
const config: OpenClawConfig = {
models: {
providers: {
openai: {
baseUrl: "https://api.openai.com/v1",
apiKey: { source: "env", id: "OPENAI_API_KEY" },
models: [],
},
},
},
skills: {
entries: {
"review-pr": {
enabled: true,
apiKey: { source: "env", id: "REVIEW_SKILL_API_KEY" },
},
},
},
};
const snapshot = await prepareSecretsRuntimeSnapshot({
config,
env: {
OPENAI_API_KEY: "sk-env-openai",
GITHUB_TOKEN: "ghp-env-token",
REVIEW_SKILL_API_KEY: "sk-skill-ref",
},
agentDirs: ["/tmp/openclaw-agent-main"],
loadAuthStore: () => ({
version: 1,
profiles: {
"openai:default": {
type: "api_key",
provider: "openai",
key: "old-openai",
keyRef: { source: "env", id: "OPENAI_API_KEY" },
},
"github-copilot:default": {
type: "token",
provider: "github-copilot",
token: "old-gh",
tokenRef: { source: "env", id: "GITHUB_TOKEN" },
},
},
}),
});
expect(snapshot.config.models?.providers?.openai?.apiKey).toBe("sk-env-openai");
expect(snapshot.config.skills?.entries?.["review-pr"]?.apiKey).toBe("sk-skill-ref");
expect(snapshot.warnings).toHaveLength(2);
expect(snapshot.authStores[0]?.store.profiles["openai:default"]).toMatchObject({
type: "api_key",
key: "sk-env-openai",
});
expect(snapshot.authStores[0]?.store.profiles["github-copilot:default"]).toMatchObject({
type: "token",
token: "ghp-env-token",
});
});
it("resolves file refs via sops json payload", async () => {
runExecMock.mockResolvedValueOnce({
stdout: JSON.stringify({
providers: {
openai: {
apiKey: "sk-from-sops",
},
},
}),
stderr: "",
});
const config: OpenClawConfig = {
secrets: {
sources: {
file: {
type: "sops",
path: "~/.openclaw/secrets.enc.json",
timeoutMs: 7000,
},
},
},
models: {
providers: {
openai: {
baseUrl: "https://api.openai.com/v1",
apiKey: { source: "file", id: "/providers/openai/apiKey" },
models: [],
},
},
},
};
const snapshot = await prepareSecretsRuntimeSnapshot({
config,
agentDirs: ["/tmp/openclaw-agent-main"],
loadAuthStore: () => ({ version: 1, profiles: {} }),
});
expect(snapshot.config.models?.providers?.openai?.apiKey).toBe("sk-from-sops");
expect(runExecMock).toHaveBeenCalledWith(
"sops",
["--decrypt", "--output-type", "json", expect.stringContaining("secrets.enc.json")],
{
timeoutMs: 7000,
maxBuffer: 10 * 1024 * 1024,
},
);
});
it("activates runtime snapshots for loadConfig and ensureAuthProfileStore", async () => {
const prepared = await prepareSecretsRuntimeSnapshot({
config: {
models: {
providers: {
openai: {
baseUrl: "https://api.openai.com/v1",
apiKey: { source: "env", id: "OPENAI_API_KEY" },
models: [],
},
},
},
},
env: { OPENAI_API_KEY: "sk-runtime" },
agentDirs: ["/tmp/openclaw-agent-main"],
loadAuthStore: () => ({
version: 1,
profiles: {
"openai:default": {
type: "api_key",
provider: "openai",
keyRef: { source: "env", id: "OPENAI_API_KEY" },
},
},
}),
});
activateSecretsRuntimeSnapshot(prepared);
expect(loadConfig().models?.providers?.openai?.apiKey).toBe("sk-runtime");
const store = ensureAuthProfileStore("/tmp/openclaw-agent-main");
expect(store.profiles["openai:default"]).toMatchObject({
type: "api_key",
key: "sk-runtime",
});
});
});

337
src/secrets/runtime.ts Normal file
View File

@@ -0,0 +1,337 @@
import { resolveOpenClawAgentDir } from "../agents/agent-paths.js";
import { listAgentIds, resolveAgentDir } from "../agents/agent-scope.js";
import type { AuthProfileCredential, AuthProfileStore } from "../agents/auth-profiles.js";
import {
clearRuntimeAuthProfileStoreSnapshots,
loadAuthProfileStoreForRuntime,
replaceRuntimeAuthProfileStoreSnapshots,
} from "../agents/auth-profiles.js";
import {
clearRuntimeConfigSnapshot,
setRuntimeConfigSnapshot,
type OpenClawConfig,
} from "../config/config.js";
import { isSecretRef, type SecretRef } from "../config/types.secrets.js";
import { resolveUserPath } from "../utils.js";
import { readJsonPointer } from "./json-pointer.js";
import { decryptSopsJsonFile, DEFAULT_SOPS_TIMEOUT_MS } from "./sops.js";
type SecretResolverWarningCode = "SECRETS_REF_OVERRIDES_PLAINTEXT";
export type SecretResolverWarning = {
code: SecretResolverWarningCode;
path: string;
message: string;
};
export type PreparedSecretsRuntimeSnapshot = {
sourceConfig: OpenClawConfig;
config: OpenClawConfig;
authStores: Array<{ agentDir: string; store: AuthProfileStore }>;
warnings: SecretResolverWarning[];
};
type ResolverContext = {
config: OpenClawConfig;
env: NodeJS.ProcessEnv;
fileSecretsPromise: Promise<unknown> | null;
};
type ProviderLike = {
apiKey?: unknown;
};
type SkillEntryLike = {
apiKey?: unknown;
};
type GoogleChatAccountLike = {
serviceAccount?: unknown;
serviceAccountRef?: unknown;
accounts?: Record<string, unknown>;
};
type ApiKeyCredentialLike = AuthProfileCredential & {
type: "api_key";
key?: string;
keyRef?: unknown;
};
type TokenCredentialLike = AuthProfileCredential & {
type: "token";
token?: string;
tokenRef?: unknown;
};
let activeSnapshot: PreparedSecretsRuntimeSnapshot | null = null;
function cloneSnapshot(snapshot: PreparedSecretsRuntimeSnapshot): PreparedSecretsRuntimeSnapshot {
return {
sourceConfig: structuredClone(snapshot.sourceConfig),
config: structuredClone(snapshot.config),
authStores: snapshot.authStores.map((entry) => ({
agentDir: entry.agentDir,
store: structuredClone(entry.store),
})),
warnings: snapshot.warnings.map((warning) => ({ ...warning })),
};
}
function isRecord(value: unknown): value is Record<string, unknown> {
return typeof value === "object" && value !== null && !Array.isArray(value);
}
function isNonEmptyString(value: unknown): value is string {
return typeof value === "string" && value.trim().length > 0;
}
async function decryptSopsFile(config: OpenClawConfig): Promise<unknown> {
const fileSource = config.secrets?.sources?.file;
if (!fileSource) {
throw new Error(
`Secret reference source "file" is not configured. Configure secrets.sources.file first.`,
);
}
if (fileSource.type !== "sops") {
throw new Error(`Unsupported secrets.sources.file.type "${String(fileSource.type)}".`);
}
const resolvedPath = resolveUserPath(fileSource.path);
const timeoutMs =
typeof fileSource.timeoutMs === "number" && Number.isFinite(fileSource.timeoutMs)
? Math.max(1, Math.floor(fileSource.timeoutMs))
: DEFAULT_SOPS_TIMEOUT_MS;
return await decryptSopsJsonFile({
path: resolvedPath,
timeoutMs,
missingBinaryMessage:
"sops binary not found in PATH. Install sops >= 3.9.0 or disable secrets.sources.file.",
});
}
async function resolveSecretRefValue(ref: SecretRef, context: ResolverContext): Promise<unknown> {
const id = ref.id.trim();
if (!id) {
throw new Error(`Secret reference id is empty.`);
}
if (ref.source === "env") {
const envValue = context.env[id];
if (!isNonEmptyString(envValue)) {
throw new Error(`Environment variable "${id}" is missing or empty.`);
}
return envValue;
}
if (ref.source === "file") {
context.fileSecretsPromise ??= decryptSopsFile(context.config);
const payload = await context.fileSecretsPromise;
return readJsonPointer(payload, id, { onMissing: "throw" });
}
throw new Error(`Unsupported secret source "${String((ref as { source?: unknown }).source)}".`);
}
async function resolveGoogleChatServiceAccount(
target: GoogleChatAccountLike,
path: string,
context: ResolverContext,
warnings: SecretResolverWarning[],
): Promise<void> {
const explicitRef = isSecretRef(target.serviceAccountRef) ? target.serviceAccountRef : null;
const inlineRef = isSecretRef(target.serviceAccount) ? target.serviceAccount : null;
const ref = explicitRef ?? inlineRef;
if (!ref) {
return;
}
if (explicitRef && target.serviceAccount !== undefined && !isSecretRef(target.serviceAccount)) {
warnings.push({
code: "SECRETS_REF_OVERRIDES_PLAINTEXT",
path,
message: `${path}: serviceAccountRef is set; runtime will ignore plaintext serviceAccount.`,
});
}
target.serviceAccount = await resolveSecretRefValue(ref, context);
}
async function resolveConfigSecretRefs(params: {
config: OpenClawConfig;
context: ResolverContext;
warnings: SecretResolverWarning[];
}): Promise<OpenClawConfig> {
const resolved = structuredClone(params.config);
const providers = resolved.models?.providers as Record<string, ProviderLike> | undefined;
if (providers) {
for (const [providerId, provider] of Object.entries(providers)) {
if (!isSecretRef(provider.apiKey)) {
continue;
}
const resolvedValue = await resolveSecretRefValue(provider.apiKey, params.context);
if (!isNonEmptyString(resolvedValue)) {
throw new Error(
`models.providers.${providerId}.apiKey resolved to a non-string or empty value.`,
);
}
provider.apiKey = resolvedValue;
}
}
const skillEntries = resolved.skills?.entries as Record<string, SkillEntryLike> | undefined;
if (skillEntries) {
for (const [skillKey, entry] of Object.entries(skillEntries)) {
if (!isSecretRef(entry.apiKey)) {
continue;
}
const resolvedValue = await resolveSecretRefValue(entry.apiKey, params.context);
if (!isNonEmptyString(resolvedValue)) {
throw new Error(
`skills.entries.${skillKey}.apiKey resolved to a non-string or empty value.`,
);
}
entry.apiKey = resolvedValue;
}
}
const googleChat = resolved.channels?.googlechat as GoogleChatAccountLike | undefined;
if (googleChat) {
await resolveGoogleChatServiceAccount(
googleChat,
"channels.googlechat",
params.context,
params.warnings,
);
if (isRecord(googleChat.accounts)) {
for (const [accountId, account] of Object.entries(googleChat.accounts)) {
if (!isRecord(account)) {
continue;
}
await resolveGoogleChatServiceAccount(
account as GoogleChatAccountLike,
`channels.googlechat.accounts.${accountId}`,
params.context,
params.warnings,
);
}
}
}
return resolved;
}
async function resolveAuthStoreSecretRefs(params: {
store: AuthProfileStore;
context: ResolverContext;
warnings: SecretResolverWarning[];
agentDir: string;
}): Promise<AuthProfileStore> {
const resolvedStore = structuredClone(params.store);
for (const [profileId, profile] of Object.entries(resolvedStore.profiles)) {
if (profile.type === "api_key") {
const apiProfile = profile as ApiKeyCredentialLike;
const keyRef = isSecretRef(apiProfile.keyRef) ? apiProfile.keyRef : null;
if (keyRef && isNonEmptyString(apiProfile.key)) {
params.warnings.push({
code: "SECRETS_REF_OVERRIDES_PLAINTEXT",
path: `${params.agentDir}.auth-profiles.${profileId}.key`,
message: `auth-profiles ${profileId}: keyRef is set; runtime will ignore plaintext key.`,
});
}
if (keyRef) {
const resolvedValue = await resolveSecretRefValue(keyRef, params.context);
if (!isNonEmptyString(resolvedValue)) {
throw new Error(`auth profile "${profileId}" keyRef resolved to an empty value.`);
}
apiProfile.key = resolvedValue;
}
continue;
}
if (profile.type === "token") {
const tokenProfile = profile as TokenCredentialLike;
const tokenRef = isSecretRef(tokenProfile.tokenRef) ? tokenProfile.tokenRef : null;
if (tokenRef && isNonEmptyString(tokenProfile.token)) {
params.warnings.push({
code: "SECRETS_REF_OVERRIDES_PLAINTEXT",
path: `${params.agentDir}.auth-profiles.${profileId}.token`,
message: `auth-profiles ${profileId}: tokenRef is set; runtime will ignore plaintext token.`,
});
}
if (tokenRef) {
const resolvedValue = await resolveSecretRefValue(tokenRef, params.context);
if (!isNonEmptyString(resolvedValue)) {
throw new Error(`auth profile "${profileId}" tokenRef resolved to an empty value.`);
}
tokenProfile.token = resolvedValue;
}
}
}
return resolvedStore;
}
function collectCandidateAgentDirs(config: OpenClawConfig): string[] {
const dirs = new Set<string>();
dirs.add(resolveUserPath(resolveOpenClawAgentDir()));
for (const agentId of listAgentIds(config)) {
dirs.add(resolveUserPath(resolveAgentDir(config, agentId)));
}
return [...dirs];
}
export async function prepareSecretsRuntimeSnapshot(params: {
config: OpenClawConfig;
env?: NodeJS.ProcessEnv;
agentDirs?: string[];
loadAuthStore?: (agentDir?: string) => AuthProfileStore;
}): Promise<PreparedSecretsRuntimeSnapshot> {
const warnings: SecretResolverWarning[] = [];
const context: ResolverContext = {
config: params.config,
env: params.env ?? process.env,
fileSecretsPromise: null,
};
const resolvedConfig = await resolveConfigSecretRefs({
config: params.config,
context,
warnings,
});
const loadAuthStore = params.loadAuthStore ?? loadAuthProfileStoreForRuntime;
const candidateDirs = params.agentDirs?.length
? [...new Set(params.agentDirs.map((entry) => resolveUserPath(entry)))]
: collectCandidateAgentDirs(resolvedConfig);
const authStores: Array<{ agentDir: string; store: AuthProfileStore }> = [];
for (const agentDir of candidateDirs) {
const rawStore = loadAuthStore(agentDir);
const resolvedStore = await resolveAuthStoreSecretRefs({
store: rawStore,
context,
warnings,
agentDir,
});
authStores.push({ agentDir, store: resolvedStore });
}
return {
sourceConfig: structuredClone(params.config),
config: resolvedConfig,
authStores,
warnings,
};
}
export function activateSecretsRuntimeSnapshot(snapshot: PreparedSecretsRuntimeSnapshot): void {
const next = cloneSnapshot(snapshot);
setRuntimeConfigSnapshot(next.config, next.sourceConfig);
replaceRuntimeAuthProfileStoreSnapshots(next.authStores);
activeSnapshot = next;
}
export function getActiveSecretsRuntimeSnapshot(): PreparedSecretsRuntimeSnapshot | null {
return activeSnapshot ? cloneSnapshot(activeSnapshot) : null;
}
export function clearSecretsRuntimeSnapshot(): void {
activeSnapshot = null;
clearRuntimeConfigSnapshot();
clearRuntimeAuthProfileStoreSnapshots();
}

120
src/secrets/sops.ts Normal file
View File

@@ -0,0 +1,120 @@
import crypto from "node:crypto";
import fs from "node:fs";
import path from "node:path";
import { runExec } from "../process/exec.js";
export const DEFAULT_SOPS_TIMEOUT_MS = 5_000;
const MAX_SOPS_OUTPUT_BYTES = 10 * 1024 * 1024;
function normalizeTimeoutMs(value: number | undefined): number {
if (typeof value === "number" && Number.isFinite(value)) {
return Math.max(1, Math.floor(value));
}
return DEFAULT_SOPS_TIMEOUT_MS;
}
function isTimeoutError(message: string | undefined): boolean {
return typeof message === "string" && message.toLowerCase().includes("timed out");
}
type SopsErrorContext = {
missingBinaryMessage: string;
operationLabel: string;
};
function toSopsError(err: unknown, params: SopsErrorContext): Error {
const error = err as NodeJS.ErrnoException & { message?: string };
if (error.code === "ENOENT") {
return new Error(params.missingBinaryMessage, { cause: err });
}
return new Error(`${params.operationLabel}: ${String(error.message ?? err)}`, {
cause: err,
});
}
function ensureDirForFile(filePath: string): void {
fs.mkdirSync(path.dirname(filePath), { recursive: true, mode: 0o700 });
}
export async function decryptSopsJsonFile(params: {
path: string;
timeoutMs?: number;
missingBinaryMessage: string;
}): Promise<unknown> {
const timeoutMs = normalizeTimeoutMs(params.timeoutMs);
try {
const { stdout } = await runExec("sops", ["--decrypt", "--output-type", "json", params.path], {
timeoutMs,
maxBuffer: MAX_SOPS_OUTPUT_BYTES,
});
return JSON.parse(stdout) as unknown;
} catch (err) {
const error = err as NodeJS.ErrnoException & { message?: string };
if (isTimeoutError(error.message)) {
throw new Error(`sops decrypt timed out after ${timeoutMs}ms for ${params.path}.`, {
cause: err,
});
}
throw toSopsError(err, {
missingBinaryMessage: params.missingBinaryMessage,
operationLabel: `sops decrypt failed for ${params.path}`,
});
}
}
export async function encryptSopsJsonFile(params: {
path: string;
payload: Record<string, unknown>;
timeoutMs?: number;
missingBinaryMessage: string;
}): Promise<void> {
ensureDirForFile(params.path);
const timeoutMs = normalizeTimeoutMs(params.timeoutMs);
const tmpPlain = path.join(
path.dirname(params.path),
`${path.basename(params.path)}.${process.pid}.${crypto.randomUUID()}.plain.tmp`,
);
const tmpEncrypted = path.join(
path.dirname(params.path),
`${path.basename(params.path)}.${process.pid}.${crypto.randomUUID()}.enc.tmp`,
);
fs.writeFileSync(tmpPlain, `${JSON.stringify(params.payload, null, 2)}\n`, "utf8");
fs.chmodSync(tmpPlain, 0o600);
try {
await runExec(
"sops",
[
"--encrypt",
"--input-type",
"json",
"--output-type",
"json",
"--output",
tmpEncrypted,
tmpPlain,
],
{
timeoutMs,
maxBuffer: MAX_SOPS_OUTPUT_BYTES,
},
);
fs.renameSync(tmpEncrypted, params.path);
fs.chmodSync(params.path, 0o600);
} catch (err) {
const error = err as NodeJS.ErrnoException & { message?: string };
if (isTimeoutError(error.message)) {
throw new Error(`sops encrypt timed out after ${timeoutMs}ms for ${params.path}.`, {
cause: err,
});
}
throw toSopsError(err, {
missingBinaryMessage: params.missingBinaryMessage,
operationLabel: `sops encrypt failed for ${params.path}`,
});
} finally {
fs.rmSync(tmpPlain, { force: true });
fs.rmSync(tmpEncrypted, { force: true });
}
}

View File

@@ -26,45 +26,11 @@ describe("roleScopesAllow", () => {
).toBe(true);
});
it("treats operator.write as satisfied by write/admin scopes", () => {
it("keeps non-read operator scopes explicit", () => {
expect(
roleScopesAllow({
role: "operator",
requestedScopes: ["operator.write"],
allowedScopes: ["operator.write"],
}),
).toBe(true);
expect(
roleScopesAllow({
role: "operator",
requestedScopes: ["operator.write"],
allowedScopes: ["operator.admin"],
}),
).toBe(true);
});
it("treats operator.approvals/operator.pairing as satisfied by operator.admin", () => {
expect(
roleScopesAllow({
role: "operator",
requestedScopes: ["operator.approvals"],
allowedScopes: ["operator.admin"],
}),
).toBe(true);
expect(
roleScopesAllow({
role: "operator",
requestedScopes: ["operator.pairing"],
allowedScopes: ["operator.admin"],
}),
).toBe(true);
});
it("does not treat operator.admin as satisfying non-operator scopes", () => {
expect(
roleScopesAllow({
role: "operator",
requestedScopes: ["system.run"],
allowedScopes: ["operator.admin"],
}),
).toBe(false);

View File

@@ -2,7 +2,6 @@ const OPERATOR_ROLE = "operator";
const OPERATOR_ADMIN_SCOPE = "operator.admin";
const OPERATOR_READ_SCOPE = "operator.read";
const OPERATOR_WRITE_SCOPE = "operator.write";
const OPERATOR_SCOPE_PREFIX = "operator.";
function normalizeScopeList(scopes: readonly string[]): string[] {
const out = new Set<string>();
@@ -16,14 +15,12 @@ function normalizeScopeList(scopes: readonly string[]): string[] {
}
function operatorScopeSatisfied(requestedScope: string, granted: Set<string>): boolean {
if (granted.has(OPERATOR_ADMIN_SCOPE) && requestedScope.startsWith(OPERATOR_SCOPE_PREFIX)) {
return true;
}
if (requestedScope === OPERATOR_READ_SCOPE) {
return granted.has(OPERATOR_READ_SCOPE) || granted.has(OPERATOR_WRITE_SCOPE);
}
if (requestedScope === OPERATOR_WRITE_SCOPE) {
return granted.has(OPERATOR_WRITE_SCOPE);
return (
granted.has(OPERATOR_READ_SCOPE) ||
granted.has(OPERATOR_WRITE_SCOPE) ||
granted.has(OPERATOR_ADMIN_SCOPE)
);
}
return granted.has(requestedScope);
}

View File

@@ -370,62 +370,6 @@ describe("Slack native command argument menus", () => {
harness.postEphemeral.mockClear();
});
it("registers options handlers without losing app receiver binding", async () => {
const commands = new Map<string, (args: unknown) => Promise<void>>();
const actions = new Map<string, (args: unknown) => Promise<void>>();
const options = new Map<string, (args: unknown) => Promise<void>>();
const postEphemeral = vi.fn().mockResolvedValue({ ok: true });
const app = {
client: { chat: { postEphemeral } },
command: (name: string, handler: (args: unknown) => Promise<void>) => {
commands.set(name, handler);
},
action: (id: string, handler: (args: unknown) => Promise<void>) => {
actions.set(id, handler);
},
options: function (this: unknown, id: string, handler: (args: unknown) => Promise<void>) {
expect(this).toBe(app);
options.set(id, handler);
},
};
const ctx = {
cfg: { commands: { native: true, nativeSkills: false } },
runtime: {},
botToken: "bot-token",
botUserId: "bot",
teamId: "T1",
allowFrom: ["*"],
dmEnabled: true,
dmPolicy: "open",
groupDmEnabled: false,
groupDmChannels: [],
defaultRequireMention: true,
groupPolicy: "open",
useAccessGroups: false,
channelsConfig: undefined,
slashCommand: {
enabled: true,
name: "openclaw",
ephemeral: true,
sessionPrefix: "slack:slash",
},
textLimit: 4000,
app,
isChannelAllowed: () => true,
resolveChannelName: async () => ({ name: "dm", type: "im" }),
resolveUserName: async () => ({ name: "Ada" }),
} as unknown;
const account = {
accountId: "acct",
config: { commands: { native: true, nativeSkills: false } },
} as unknown;
await registerCommands(ctx, account);
expect(commands.size).toBeGreaterThan(0);
expect(actions.has("openclaw_cmdarg")).toBe(true);
expect(options.has("openclaw_cmdarg")).toBe(true);
});
it("shows a button menu when required args are omitted", async () => {
const { respond } = await runCommandHandler(usageHandler);
const actions = expectArgMenuLayout(respond);

View File

@@ -734,19 +734,21 @@ export async function registerSlackMonitorSlashCommands(params: {
}
const registerArgOptions = () => {
const appWithOptions = ctx.app as unknown as {
options?: (
actionId: string,
handler: (args: {
ack: (payload: { options: unknown[] }) => Promise<void>;
body: unknown;
}) => Promise<void>,
) => void;
};
if (typeof appWithOptions.options !== "function") {
const optionsHandler = (
ctx.app as unknown as {
options?: (
actionId: string,
handler: (args: {
ack: (payload: { options: unknown[] }) => Promise<void>;
body: unknown;
}) => Promise<void>,
) => void;
}
).options;
if (typeof optionsHandler !== "function") {
return;
}
appWithOptions.options(SLACK_COMMAND_ARG_ACTION_ID, async ({ ack, body }) => {
optionsHandler(SLACK_COMMAND_ARG_ACTION_ID, async ({ ack, body }) => {
const typedBody = body as {
value?: string;
user?: { id?: string };

View File

@@ -2,58 +2,6 @@ import { describe, expect, it, vi } from "vitest";
import { createCommandHandlers } from "./tui-command-handlers.js";
describe("tui command handlers", () => {
it("renders the sending indicator before chat.send resolves", async () => {
let resolveSend: ((value: { runId: string }) => void) | null = null;
const sendChat = vi.fn(
() =>
new Promise<{ runId: string }>((resolve) => {
resolveSend = resolve;
}),
);
const addUser = vi.fn();
const requestRender = vi.fn();
const setActivityStatus = vi.fn();
const { handleCommand } = createCommandHandlers({
client: { sendChat } as never,
chatLog: { addUser, addSystem: vi.fn() } as never,
tui: { requestRender } as never,
opts: {},
state: {
currentSessionKey: "agent:main:main",
activeChatRunId: null,
sessionInfo: {},
} as never,
deliverDefault: false,
openOverlay: vi.fn(),
closeOverlay: vi.fn(),
refreshSessionInfo: vi.fn(),
loadHistory: vi.fn(),
setSession: vi.fn(),
refreshAgents: vi.fn(),
abortActive: vi.fn(),
setActivityStatus,
formatSessionKey: vi.fn(),
applySessionInfoFromPatch: vi.fn(),
noteLocalRunId: vi.fn(),
});
const pending = handleCommand("/context");
await Promise.resolve();
expect(setActivityStatus).toHaveBeenCalledWith("sending");
const sendingOrder = setActivityStatus.mock.invocationCallOrder[0] ?? 0;
const renderOrders = requestRender.mock.invocationCallOrder;
expect(renderOrders.some((order) => order > sendingOrder)).toBe(true);
if (typeof resolveSend !== "function") {
throw new Error("expected sendChat to be pending");
}
(resolveSend as (value: { runId: string }) => void)({ runId: "r1" });
await pending;
expect(setActivityStatus).toHaveBeenCalledWith("waiting");
});
it("forwards unknown slash commands to the gateway", async () => {
const sendChat = vi.fn().mockResolvedValue({ runId: "r1" });
const addUser = vi.fn();

View File

@@ -470,7 +470,6 @@ export function createCommandHandlers(context: CommandHandlerContext) {
noteLocalRunId(runId);
state.activeChatRunId = runId;
setActivityStatus("sending");
tui.requestRender();
await client.sendChat({
sessionKey: state.currentSessionKey,
message: text,
@@ -480,7 +479,6 @@ export function createCommandHandlers(context: CommandHandlerContext) {
runId,
});
setActivityStatus("waiting");
tui.requestRender();
} catch (err) {
if (state.activeChatRunId) {
forgetLocalRunId?.(state.activeChatRunId);
@@ -488,8 +486,8 @@ export function createCommandHandlers(context: CommandHandlerContext) {
state.activeChatRunId = null;
chatLog.addSystem(`send failed: ${String(err)}`);
setActivityStatus("error");
tui.requestRender();
}
tui.requestRender();
};
return {

View File

@@ -130,32 +130,10 @@ describe("shouldEnableWindowsGitBashPasteFallback", () => {
).toBe(true);
});
it("enables fallback on macOS iTerm", () => {
expect(
shouldEnableWindowsGitBashPasteFallback({
platform: "darwin",
env: {
TERM_PROGRAM: "iTerm.app",
} as NodeJS.ProcessEnv,
}),
).toBe(true);
});
it("enables fallback on macOS Terminal.app", () => {
expect(
shouldEnableWindowsGitBashPasteFallback({
platform: "darwin",
env: {
TERM_PROGRAM: "Apple_Terminal",
} as NodeJS.ProcessEnv,
}),
).toBe(true);
});
it("disables fallback outside Windows", () => {
expect(
shouldEnableWindowsGitBashPasteFallback({
platform: "linux",
platform: "darwin",
env: {
MSYSTEM: "MINGW64",
} as NodeJS.ProcessEnv,

View File

@@ -84,24 +84,13 @@ export function shouldEnableWindowsGitBashPasteFallback(params?: {
env?: NodeJS.ProcessEnv;
}): boolean {
const platform = params?.platform ?? process.platform;
const env = params?.env ?? process.env;
const termProgram = (env.TERM_PROGRAM ?? "").toLowerCase();
// Some macOS terminals emit multiline paste as rapid single-line submits.
// Enable burst coalescing so pasted blocks stay as one user message.
if (platform === "darwin") {
if (termProgram.includes("iterm") || termProgram.includes("apple_terminal")) {
return true;
}
return false;
}
if (platform !== "win32") {
return false;
}
const env = params?.env ?? process.env;
const msystem = (env.MSYSTEM ?? "").toUpperCase();
const shell = env.SHELL ?? "";
const termProgram = (env.TERM_PROGRAM ?? "").toLowerCase();
if (msystem.startsWith("MINGW") || msystem.startsWith("MSYS")) {
return true;
}

View File

@@ -372,13 +372,6 @@
border-color: rgba(255, 255, 255, 0.2);
}
/* Ensure chat toolbar toggles have a clearly visible active state. */
.chat-controls .btn--icon.active {
border-color: var(--accent);
background: var(--accent-subtle);
color: var(--accent);
}
/* Light theme icon button overrides */
:root[data-theme="light"] .btn--icon {
background: #ffffff;
@@ -393,13 +386,6 @@
color: var(--text);
}
:root[data-theme="light"] .chat-controls .btn--icon.active {
border-color: var(--accent);
background: var(--accent-subtle);
color: var(--accent);
box-shadow: 0 0 0 1px var(--accent-subtle);
}
.btn--icon svg {
display: block;
width: 18px;

View File

@@ -542,12 +542,6 @@
background: var(--bg-hover);
}
:root[data-theme="light"] .btn.active {
border-color: var(--accent);
background: var(--accent-subtle);
color: var(--accent);
}
:root[data-theme="light"] .btn.primary {
background: var(--accent);
border-color: var(--accent);