feat: make compaction timeout configurable via agents.defaults.compaction.timeoutSeconds (#46889)

* feat: make compaction timeout configurable via agents.defaults.compaction.timeoutSeconds

The hardcoded 5-minute (300s) compaction timeout causes large sessions
to enter a death spiral where compaction repeatedly fails and the
session grows indefinitely. This adds agents.defaults.compaction.timeoutSeconds
to allow operators to override the compaction safety timeout.

Default raised to 900s (15min) which is sufficient for sessions up to
~400k tokens. The resolved timeout is also used for the session write
lock duration so locks don't expire before compaction completes.

Fixes #38233

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* test: add resolveCompactionTimeoutMs tests

Cover config resolution edge cases: undefined config, missing
compaction section, valid seconds, fractional values, zero,
negative, NaN, and Infinity.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: add timeoutSeconds to compaction Zod schema

The compaction object schema uses .strict(), so setting the new
timeoutSeconds config option would fail validation at startup.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: enforce integer constraint on compaction timeoutSeconds schema

Prevents sub-second values like 0.5 which would floor to 0ms and
cause immediate compaction timeout. Matches pattern of other
integer timeout fields in the schema.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: clamp compaction timeout to Node timer-safe maximum

Values above ~2.1B ms overflow Node's setTimeout to 1ms, causing
immediate timeout. Clamp to MAX_SAFE_TIMEOUT_MS matching the
pattern in agents/timeout.ts.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: add FIELD_LABELS entry for compaction timeoutSeconds

Maintains label/help parity invariant enforced by
schema.help.quality.test.ts.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: align compaction timeouts with abort handling

* fix: land compaction timeout handling (#46889) (thanks @asyncjason)

---------

Co-authored-by: Jason Separovic <jason@wilma.dog>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Ayaan Zaidi <hi@obviy.us>
This commit is contained in:
Jason 2026-03-14 23:34:48 -07:00 committed by GitHub
parent 8e04d1fe15
commit f77a684131
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
13 changed files with 330 additions and 32 deletions

View File

@ -27,6 +27,7 @@ Docs: https://docs.openclaw.ai
- Zalo Personal/group gating: stop reapplying `dmPolicy.allowFrom` as a sender gate for already-allowlisted groups when `groupAllowFrom` is unset, so any member of an allowed group can trigger replies while DMs stay restricted. (#40146)
- Plugins/install precedence: keep bundled plugins ahead of auto-discovered globals by default, but let an explicitly installed plugin record win its own duplicate-id tie so installed channel plugins load from `~/.openclaw/extensions` after `openclaw plugins install`.
- macOS/canvas actions: keep unattended local agent actions on trusted in-app canvas surfaces only, and stop exposing the deep-link fallback key to arbitrary page scripts. Thanks @vincentkoc.
- Agents/compaction: extend the enclosing run deadline once while compaction is actively in flight, and abort the underlying SDK compaction on timeout/cancel so large-session compactions stop freezing mid-run. (#46889) Thanks @asyncjason.
### Fixes

View File

@ -2,6 +2,7 @@ import { afterEach, describe, expect, it, vi } from "vitest";
import {
compactWithSafetyTimeout,
EMBEDDED_COMPACTION_TIMEOUT_MS,
resolveCompactionTimeoutMs,
} from "./pi-embedded-runner/compaction-safety-timeout.js";
describe("compactWithSafetyTimeout", () => {
@ -42,4 +43,99 @@ describe("compactWithSafetyTimeout", () => {
).rejects.toBe(error);
expect(vi.getTimerCount()).toBe(0);
});
it("calls onCancel when compaction times out", async () => {
vi.useFakeTimers();
const onCancel = vi.fn();
const compactPromise = compactWithSafetyTimeout(() => new Promise<never>(() => {}), 30, {
onCancel,
});
const timeoutAssertion = expect(compactPromise).rejects.toThrow("Compaction timed out");
await vi.advanceTimersByTimeAsync(30);
await timeoutAssertion;
expect(onCancel).toHaveBeenCalledTimes(1);
expect(vi.getTimerCount()).toBe(0);
});
it("aborts early on external abort signal and calls onCancel once", async () => {
vi.useFakeTimers();
const controller = new AbortController();
const onCancel = vi.fn();
const reason = new Error("request timed out");
const compactPromise = compactWithSafetyTimeout(() => new Promise<never>(() => {}), 100, {
abortSignal: controller.signal,
onCancel,
});
const abortAssertion = expect(compactPromise).rejects.toBe(reason);
controller.abort(reason);
await abortAssertion;
expect(onCancel).toHaveBeenCalledTimes(1);
expect(vi.getTimerCount()).toBe(0);
});
});
describe("resolveCompactionTimeoutMs", () => {
it("returns default when config is undefined", () => {
expect(resolveCompactionTimeoutMs(undefined)).toBe(EMBEDDED_COMPACTION_TIMEOUT_MS);
});
it("returns default when compaction config is missing", () => {
expect(resolveCompactionTimeoutMs({ agents: { defaults: {} } })).toBe(
EMBEDDED_COMPACTION_TIMEOUT_MS,
);
});
it("returns default when timeoutSeconds is not set", () => {
expect(
resolveCompactionTimeoutMs({ agents: { defaults: { compaction: { mode: "safeguard" } } } }),
).toBe(EMBEDDED_COMPACTION_TIMEOUT_MS);
});
it("converts timeoutSeconds to milliseconds", () => {
expect(
resolveCompactionTimeoutMs({
agents: { defaults: { compaction: { timeoutSeconds: 1800 } } },
}),
).toBe(1_800_000);
});
it("floors fractional seconds", () => {
expect(
resolveCompactionTimeoutMs({
agents: { defaults: { compaction: { timeoutSeconds: 120.7 } } },
}),
).toBe(120_000);
});
it("returns default for zero", () => {
expect(
resolveCompactionTimeoutMs({ agents: { defaults: { compaction: { timeoutSeconds: 0 } } } }),
).toBe(EMBEDDED_COMPACTION_TIMEOUT_MS);
});
it("returns default for negative values", () => {
expect(
resolveCompactionTimeoutMs({ agents: { defaults: { compaction: { timeoutSeconds: -5 } } } }),
).toBe(EMBEDDED_COMPACTION_TIMEOUT_MS);
});
it("returns default for NaN", () => {
expect(
resolveCompactionTimeoutMs({
agents: { defaults: { compaction: { timeoutSeconds: NaN } } },
}),
).toBe(EMBEDDED_COMPACTION_TIMEOUT_MS);
});
it("returns default for Infinity", () => {
expect(
resolveCompactionTimeoutMs({
agents: { defaults: { compaction: { timeoutSeconds: Infinity } } },
}),
).toBe(EMBEDDED_COMPACTION_TIMEOUT_MS);
});
});

View File

@ -14,6 +14,7 @@ const {
resolveMemorySearchConfigMock,
resolveSessionAgentIdMock,
estimateTokensMock,
sessionAbortCompactionMock,
} = vi.hoisted(() => {
const contextEngineCompactMock = vi.fn(async () => ({
ok: true as boolean,
@ -65,6 +66,7 @@ const {
})),
resolveSessionAgentIdMock: vi.fn(() => "main"),
estimateTokensMock: vi.fn((_message?: unknown) => 10),
sessionAbortCompactionMock: vi.fn(),
};
});
@ -121,6 +123,7 @@ vi.mock("@mariozechner/pi-coding-agent", () => {
session.messages.splice(1);
return await sessionCompactImpl();
}),
abortCompaction: sessionAbortCompactionMock,
dispose: vi.fn(),
};
return { session };
@ -151,6 +154,7 @@ vi.mock("../models-config.js", () => ({
}));
vi.mock("../model-auth.js", () => ({
applyLocalNoAuthHeaderOverride: vi.fn((model: unknown) => model),
getApiKeyForModel: vi.fn(async () => ({ apiKey: "test", mode: "env" })),
resolveModelAuthMode: vi.fn(() => "env"),
}));
@ -420,6 +424,7 @@ describe("compactEmbeddedPiSessionDirect hooks", () => {
resolveSessionAgentIdMock.mockReturnValue("main");
estimateTokensMock.mockReset();
estimateTokensMock.mockReturnValue(10);
sessionAbortCompactionMock.mockReset();
unregisterApiProviders(getCustomApiRegistrySourceId("ollama"));
});
@ -772,6 +777,24 @@ describe("compactEmbeddedPiSessionDirect hooks", () => {
expect(result.ok).toBe(true);
});
it("aborts in-flight compaction when the caller abort signal fires", async () => {
const controller = new AbortController();
sessionCompactImpl.mockImplementationOnce(() => new Promise<never>(() => {}));
const resultPromise = compactEmbeddedPiSessionDirect(
directCompactionArgs({
abortSignal: controller.signal,
}),
);
controller.abort(new Error("request timed out"));
const result = await resultPromise;
expect(result.ok).toBe(false);
expect(result.reason).toContain("request timed out");
expect(sessionAbortCompactionMock).toHaveBeenCalledTimes(1);
});
});
describe("compactEmbeddedPiSession hooks (ownsCompaction engine)", () => {

View File

@ -76,7 +76,7 @@ import {
import { resolveTranscriptPolicy } from "../transcript-policy.js";
import {
compactWithSafetyTimeout,
EMBEDDED_COMPACTION_TIMEOUT_MS,
resolveCompactionTimeoutMs,
} from "./compaction-safety-timeout.js";
import { buildEmbeddedExtensionFactories } from "./extensions.js";
import {
@ -143,6 +143,7 @@ export type CompactEmbeddedPiSessionParams = {
enqueue?: typeof enqueueCommand;
extraSystemPrompt?: string;
ownerNumbers?: string[];
abortSignal?: AbortSignal;
};
type CompactionMessageMetrics = {
@ -687,10 +688,11 @@ export async function compactEmbeddedPiSessionDirect(
});
const systemPromptOverride = createSystemPromptOverride(appendPrompt);
const compactionTimeoutMs = resolveCompactionTimeoutMs(params.config);
const sessionLock = await acquireSessionWriteLock({
sessionFile: params.sessionFile,
maxHoldMs: resolveSessionLockMaxHoldFromTimeout({
timeoutMs: EMBEDDED_COMPACTION_TIMEOUT_MS,
timeoutMs: compactionTimeoutMs,
}),
});
try {
@ -915,8 +917,15 @@ export async function compactEmbeddedPiSessionDirect(
// If token estimation throws on a malformed message, fall back to 0 so
// the sanity check below becomes a no-op instead of crashing compaction.
}
const result = await compactWithSafetyTimeout(() =>
session.compact(params.customInstructions),
const result = await compactWithSafetyTimeout(
() => session.compact(params.customInstructions),
compactionTimeoutMs,
{
abortSignal: params.abortSignal,
onCancel: () => {
session.abortCompaction();
},
},
);
await runPostCompactionSideEffects({
config: params.config,

View File

@ -1,10 +1,88 @@
import type { OpenClawConfig } from "../../config/config.js";
import { withTimeout } from "../../node-host/with-timeout.js";
export const EMBEDDED_COMPACTION_TIMEOUT_MS = 300_000;
export const EMBEDDED_COMPACTION_TIMEOUT_MS = 900_000;
const MAX_SAFE_TIMEOUT_MS = 2_147_000_000;
function createAbortError(signal: AbortSignal): Error {
const reason = "reason" in signal ? signal.reason : undefined;
if (reason instanceof Error) {
return reason;
}
const err = reason ? new Error("aborted", { cause: reason }) : new Error("aborted");
err.name = "AbortError";
return err;
}
export function resolveCompactionTimeoutMs(cfg?: OpenClawConfig): number {
const raw = cfg?.agents?.defaults?.compaction?.timeoutSeconds;
if (typeof raw === "number" && Number.isFinite(raw) && raw > 0) {
return Math.min(Math.floor(raw) * 1000, MAX_SAFE_TIMEOUT_MS);
}
return EMBEDDED_COMPACTION_TIMEOUT_MS;
}
export async function compactWithSafetyTimeout<T>(
compact: () => Promise<T>,
timeoutMs: number = EMBEDDED_COMPACTION_TIMEOUT_MS,
opts?: {
abortSignal?: AbortSignal;
onCancel?: () => void;
},
): Promise<T> {
return await withTimeout(() => compact(), timeoutMs, "Compaction");
let canceled = false;
const cancel = () => {
if (canceled) {
return;
}
canceled = true;
opts?.onCancel?.();
};
return await withTimeout(
async (timeoutSignal) => {
let timeoutListener: (() => void) | undefined;
let externalAbortListener: (() => void) | undefined;
let externalAbortPromise: Promise<never> | undefined;
const abortSignal = opts?.abortSignal;
if (timeoutSignal) {
timeoutListener = () => {
cancel();
};
timeoutSignal.addEventListener("abort", timeoutListener, { once: true });
}
if (abortSignal) {
if (abortSignal.aborted) {
cancel();
throw createAbortError(abortSignal);
}
externalAbortPromise = new Promise((_, reject) => {
externalAbortListener = () => {
cancel();
reject(createAbortError(abortSignal));
};
abortSignal.addEventListener("abort", externalAbortListener, { once: true });
});
}
try {
if (externalAbortPromise) {
return await Promise.race([compact(), externalAbortPromise]);
}
return await compact();
} finally {
if (timeoutListener) {
timeoutSignal?.removeEventListener("abort", timeoutListener);
}
if (externalAbortListener) {
abortSignal?.removeEventListener("abort", externalAbortListener);
}
}
},
timeoutMs,
"Compaction",
);
}

View File

@ -97,6 +97,7 @@ import { DEFAULT_BOOTSTRAP_FILENAME } from "../../workspace.js";
import { isRunnerAbortError } from "../abort.js";
import { appendCacheTtlTimestamp, isCacheTtlEligibleProvider } from "../cache-ttl.js";
import type { CompactEmbeddedPiSessionParams } from "../compact.js";
import { resolveCompactionTimeoutMs } from "../compaction-safety-timeout.js";
import { buildEmbeddedExtensionFactories } from "../extensions.js";
import { applyExtraParamsToAgent } from "../extra-params.js";
import {
@ -130,6 +131,7 @@ import { describeUnknownError, mapThinkingLevel } from "../utils.js";
import { flushPendingToolResultsAfterIdle } from "../wait-for-idle-before-flush.js";
import { waitForCompactionRetryWithAggregateTimeout } from "./compaction-retry-aggregate-timeout.js";
import {
resolveRunTimeoutDuringCompaction,
selectCompactionTimeoutSnapshot,
shouldFlagCompactionTimeout,
} from "./compaction-timeout.js";
@ -2150,6 +2152,20 @@ export async function runEmbeddedAttempt(
err.name = "AbortError";
return err;
};
const abortCompaction = () => {
if (!activeSession.isCompacting) {
return;
}
try {
activeSession.abortCompaction();
} catch (err) {
if (!isProbeSession) {
log.warn(
`embedded run abortCompaction failed: runId=${params.runId} sessionId=${params.sessionId} err=${String(err)}`,
);
}
}
};
const abortRun = (isTimeout = false, reason?: unknown) => {
aborted = true;
if (isTimeout) {
@ -2160,6 +2176,7 @@ export async function runEmbeddedAttempt(
} else {
runAbortController.abort(reason);
}
abortCompaction();
void activeSession.abort();
};
const abortable = <T>(promise: Promise<T>): Promise<T> => {
@ -2240,11 +2257,34 @@ export async function runEmbeddedAttempt(
let abortWarnTimer: NodeJS.Timeout | undefined;
const isProbeSession = params.sessionId?.startsWith("probe-") ?? false;
const abortTimer = setTimeout(
const compactionTimeoutMs = resolveCompactionTimeoutMs(params.config);
let abortTimer: NodeJS.Timeout | undefined;
let compactionGraceUsed = false;
const scheduleAbortTimer = (delayMs: number, reason: "initial" | "compaction-grace") => {
abortTimer = setTimeout(
() => {
const timeoutAction = resolveRunTimeoutDuringCompaction({
isCompactionPendingOrRetrying: subscription.isCompacting(),
isCompactionInFlight: activeSession.isCompacting,
graceAlreadyUsed: compactionGraceUsed,
});
if (timeoutAction === "extend") {
compactionGraceUsed = true;
if (!isProbeSession) {
log.warn(
`embedded run timeout: runId=${params.runId} sessionId=${params.sessionId} timeoutMs=${params.timeoutMs}`,
`embedded run timeout reached during compaction; extending deadline: ` +
`runId=${params.runId} sessionId=${params.sessionId} extraMs=${compactionTimeoutMs}`,
);
}
scheduleAbortTimer(compactionTimeoutMs, "compaction-grace");
return;
}
if (!isProbeSession) {
log.warn(
reason === "compaction-grace"
? `embedded run timeout after compaction grace: runId=${params.runId} sessionId=${params.sessionId} timeoutMs=${params.timeoutMs} compactionGraceMs=${compactionTimeoutMs}`
: `embedded run timeout: runId=${params.runId} sessionId=${params.sessionId} timeoutMs=${params.timeoutMs}`,
);
}
if (
@ -2270,8 +2310,10 @@ export async function runEmbeddedAttempt(
}, 10_000);
}
},
Math.max(1, params.timeoutMs),
Math.max(1, delayMs),
);
};
scheduleAbortTimer(params.timeoutMs, "initial");
let messagesSnapshot: AgentMessage[] = [];
let sessionIdUsed = activeSession.sessionId;

View File

@ -1,6 +1,7 @@
import { describe, expect, it } from "vitest";
import { castAgentMessage } from "../../test-helpers/agent-message-fixtures.js";
import {
resolveRunTimeoutDuringCompaction,
selectCompactionTimeoutSnapshot,
shouldFlagCompactionTimeout,
} from "./compaction-timeout.js";
@ -31,6 +32,36 @@ describe("compaction-timeout helpers", () => {
).toBe(false);
});
it("extends the first run timeout reached during compaction", () => {
expect(
resolveRunTimeoutDuringCompaction({
isCompactionPendingOrRetrying: false,
isCompactionInFlight: true,
graceAlreadyUsed: false,
}),
).toBe("extend");
});
it("aborts after compaction grace has already been used", () => {
expect(
resolveRunTimeoutDuringCompaction({
isCompactionPendingOrRetrying: true,
isCompactionInFlight: false,
graceAlreadyUsed: true,
}),
).toBe("abort");
});
it("aborts immediately when no compaction is active", () => {
expect(
resolveRunTimeoutDuringCompaction({
isCompactionPendingOrRetrying: false,
isCompactionInFlight: false,
graceAlreadyUsed: false,
}),
).toBe("abort");
});
it("uses pre-compaction snapshot when compaction timeout occurs", () => {
const pre = [castAgentMessage({ role: "assistant", content: "pre" })] as const;
const current = [castAgentMessage({ role: "assistant", content: "current" })] as const;

View File

@ -13,6 +13,17 @@ export function shouldFlagCompactionTimeout(signal: CompactionTimeoutSignal): bo
return signal.isCompactionPendingOrRetrying || signal.isCompactionInFlight;
}
export function resolveRunTimeoutDuringCompaction(params: {
isCompactionPendingOrRetrying: boolean;
isCompactionInFlight: boolean;
graceAlreadyUsed: boolean;
}): "extend" | "abort" {
if (!params.isCompactionPendingOrRetrying && !params.isCompactionInFlight) {
return "abort";
}
return params.graceAlreadyUsed ? "abort" : "extend";
}
export type SnapshotSelectionParams = {
timedOutDuringCompaction: boolean;
preCompactionSnapshot: AgentMessage[] | null;

View File

@ -384,6 +384,7 @@ const TARGET_KEYS = [
"agents.defaults.compaction.qualityGuard.enabled",
"agents.defaults.compaction.qualityGuard.maxRetries",
"agents.defaults.compaction.postCompactionSections",
"agents.defaults.compaction.timeoutSeconds",
"agents.defaults.compaction.model",
"agents.defaults.compaction.memoryFlush",
"agents.defaults.compaction.memoryFlush.enabled",

View File

@ -1045,6 +1045,8 @@ export const FIELD_HELP: Record<string, string> = {
'Controls post-compaction session memory reindex mode: "off", "async", or "await" (default: "async"). Use "await" for strongest freshness, "async" for lower compaction latency, and "off" only when session-memory sync is handled elsewhere.',
"agents.defaults.compaction.postCompactionSections":
'AGENTS.md H2/H3 section names re-injected after compaction so the agent reruns critical startup guidance. Leave unset to use "Session Startup"/"Red Lines" with legacy fallback to "Every Session"/"Safety"; set to [] to disable reinjection entirely.',
"agents.defaults.compaction.timeoutSeconds":
"Maximum time in seconds allowed for a single compaction operation before it is aborted (default: 900). Increase this for very large sessions that need more time to summarize, or decrease it to fail faster on unresponsive models.",
"agents.defaults.compaction.model":
"Optional provider/model override used only for compaction summarization. Set this when you want compaction to run on a different model than the session default, and leave it unset to keep using the primary agent model.",
"agents.defaults.compaction.memoryFlush":

View File

@ -474,6 +474,7 @@ export const FIELD_LABELS: Record<string, string> = {
"agents.defaults.compaction.qualityGuard.maxRetries": "Compaction Quality Guard Max Retries",
"agents.defaults.compaction.postIndexSync": "Compaction Post-Index Sync",
"agents.defaults.compaction.postCompactionSections": "Post-Compaction Context Sections",
"agents.defaults.compaction.timeoutSeconds": "Compaction Timeout (Seconds)",
"agents.defaults.compaction.model": "Compaction Model Override",
"agents.defaults.compaction.memoryFlush": "Compaction Memory Flush",
"agents.defaults.compaction.memoryFlush.enabled": "Compaction Memory Flush Enabled",

View File

@ -338,6 +338,8 @@ export type AgentCompactionConfig = {
* When set, compaction uses this model instead of the agent's primary model.
* Falls back to the primary model when unset. */
model?: string;
/** Maximum time in seconds for a single compaction operation (default: 900). */
timeoutSeconds?: number;
};
export type AgentCompactionMemoryFlushConfig = {

View File

@ -107,6 +107,7 @@ export const AgentDefaultsSchema = z
postIndexSync: z.enum(["off", "async", "await"]).optional(),
postCompactionSections: z.array(z.string()).optional(),
model: z.string().optional(),
timeoutSeconds: z.number().int().positive().optional(),
memoryFlush: z
.object({
enabled: z.boolean().optional(),