nyrn
/
openclaw
mirror of https://github.com/openclaw/openclaw.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
openclaw/src/gateway
History
Vishal Doshi e91a5b0216 fix: release stale session locks and add watchdog for hung API calls (#18060) 
When a model API call hangs indefinitely (e.g. Anthropic quota exceeded
mid-call), the gateway acquires a session .jsonl.lock but the promise
never resolves, so the try/finally block never reaches release(). Since
the owning PID is the gateway itself, stale detection cannot help —
isPidAlive() always returns true.

This commit adds four layers of defense:

1. **In-process lock watchdog** (session-write-lock.ts)
   - Track acquiredAt timestamp on each held lock
   - 60-second interval timer checks all held locks
   - Auto-releases any lock held longer than maxHoldMs (default 5 min)
   - Catches the hung-API-call case that try/finally cannot

2. **Gateway startup cleanup** (server-startup.ts)
   - On boot, scan all agent session directories for *.jsonl.lock files
   - Remove locks with dead PIDs or older than staleMs (30 min)
   - Log each cleaned lock for diagnostics

3. **openclaw doctor stale lock detection** (doctor-session-locks.ts)
   - New health check scans for .jsonl.lock files
   - Reports PID status and age of each lock found
   - In --fix mode, removes stale locks automatically

4. **Transcript error entry on API failure** (attempt.ts)
   - When promptError is set, write an error marker to the session
     transcript before releasing the lock
   - Preserves conversation history even on model API failures

Closes #18060
 		2026-02-16 23:59:22 +01:00
..
protocol Add mesh auto-planning with chat command UX and hardened auth/session behavior 2026-02-16 23:58:23 +01:00
server refactor(channels): dedupe transport and gateway test scaffolds 2026-02-16 14:59:31 +00:00
server-methods Add mesh auto-planning with chat command UX and hardened auth/session behavior 2026-02-16 23:58:23 +01:00
agent-event-assistant-text.ts refactor(gateway): dedupe assistant delta parsing 2026-02-15 19:08:47 +00:00
agent-prompt.e2e.test.ts refactor(gateway): share agent prompt builder 2026-02-14 15:39:45 +00:00
agent-prompt.ts refactor(gateway): share agent prompt builder 2026-02-14 15:39:45 +00:00
assistant-identity.test.ts chore: Enable "experimentalSortImports" in Oxfmt and reformat all imorts. 2026-02-01 10:03:47 +09:00
assistant-identity.ts feat(ui): add Agents dashboard 2026-02-02 21:31:17 -05:00
auth-rate-limit.test.ts feat(gateway): add auth rate-limiting & brute-force protection (#15035) 2026-02-13 15:32:38 +01:00
auth-rate-limit.ts feat(gateway): add auth rate-limiting & brute-force protection (#15035) 2026-02-13 15:32:38 +01:00
auth.test.ts refactor(test): dedupe trusted-proxy auth test setup 2026-02-16 18:31:37 +00:00
auth.ts refactor(gateway): share host header parsing 2026-02-15 16:15:53 +00:00
boot.test.ts refactor(channels): dedupe transport and gateway test scaffolds 2026-02-16 14:59:31 +00:00
boot.ts fix(gateway): keep boot sessions ephemeral without remapping main 2026-02-16 00:03:21 +00:00
call.test.ts refactor(test): reuse env snapshots in gateway call tests 2026-02-15 23:22:58 +00:00
call.ts fix: context overflow compaction and subagent announce improvements (#11664) (thanks @tyler6204) 2026-02-07 20:02:32 -08:00
channel-health-monitor.test.ts feat(gateway): add channel health monitor with auto-restart 2026-02-16 23:53:35 +01:00
channel-health-monitor.ts feat(gateway): add channel health monitor with auto-restart 2026-02-16 23:53:35 +01:00
chat-abort.test.ts Gateway/Control UI: preserve partial output on abort (#15026) 2026-02-15 16:55:28 -08:00
chat-abort.ts Gateway/Control UI: preserve partial output on abort (#15026) 2026-02-15 16:55:28 -08:00
chat-attachments.test.ts fix(security): reject oversized base64 before decode 2026-02-14 15:45:41 +01:00
chat-attachments.ts refactor(channels): dedupe transport and gateway test scaffolds 2026-02-16 14:59:31 +00:00
chat-sanitize.test.ts fix: hide message_id hints in web chat 2026-01-24 13:52:31 +00:00
chat-sanitize.ts refactor: dedupe chat envelope + daemon output + skills UI 2026-02-15 03:41:11 +00:00
client.e2e.test.ts perf: speed up gateway missing-tick e2e watchdog 2026-02-13 19:52:45 +00:00
client.ts fix: clear stale device-auth token on token mismatch 2026-02-16 23:54:23 +01:00
config-reload.test.ts chore: Enable "experimentalSortImports" in Oxfmt and reformat all imorts. 2026-02-01 10:03:47 +09:00
config-reload.ts fix: ignore meta field changes in config file watcher (#13460) 2026-02-12 07:55:26 -06:00
control-ui-contract.ts refactor(gateway): share Control UI bootstrap contract and CSP 2026-02-16 03:50:39 +01:00
control-ui-csp.test.ts refactor(gateway): share Control UI bootstrap contract and CSP 2026-02-16 03:50:39 +01:00
control-ui-csp.ts refactor(gateway): share Control UI bootstrap contract and CSP 2026-02-16 03:50:39 +01:00
control-ui-shared.ts chore: Enable "curly" rule to avoid single-statement if confusion/errors. 2026-01-31 16:19:20 +09:00
control-ui.http.test.ts refactor(channels): dedupe transport and gateway test scaffolds 2026-02-16 14:59:31 +00:00
control-ui.ts refactor(gateway): share Control UI bootstrap contract and CSP 2026-02-16 03:50:39 +01:00
device-auth.ts feat: enforce device-bound connect challenge 2026-01-20 13:04:19 +00:00
exec-approval-manager.ts fix(gateway): bind system.run approvals to exec approvals 2026-02-14 13:27:45 +01:00
gateway-cli-backend.live.test.ts refactor(channels): dedupe transport and gateway test scaffolds 2026-02-16 14:59:31 +00:00
gateway-config-prompts.shared.ts refactor(gateway): share tailscale prompt constants 2026-02-15 18:06:48 +00:00
gateway-misc.test.ts test(gateway): move Control UI http coverage 2026-02-16 03:50:39 +01:00
gateway-models.profiles.live.test.ts chore: Enable "experimentalSortImports" in Oxfmt and reformat all imorts. 2026-02-01 10:03:47 +09:00
gateway.e2e.test.ts refactor(channels): dedupe transport and gateway test scaffolds 2026-02-16 14:59:31 +00:00
hooks-mapping.test.ts refactor(channels): dedupe transport and gateway test scaffolds 2026-02-16 14:59:31 +00:00
hooks-mapping.ts fix(security): restrict hook transform module loading 2026-02-14 13:46:09 +01:00
hooks.test.ts refactor(channels): dedupe transport and gateway test scaffolds 2026-02-16 14:59:31 +00:00
hooks.ts fix(security): enforce bounded webhook body handling 2026-02-13 19:14:54 +01:00
http-auth-helpers.ts refactor(gateway): share bearer auth helper 2026-02-15 04:40:04 +00:00
http-common.ts fix(security): enforce bounded webhook body handling 2026-02-13 19:14:54 +01:00
http-endpoint-helpers.test.ts refactor(gateway): dedupe json endpoint prelude 2026-02-15 13:24:37 +00:00
http-endpoint-helpers.ts refactor(gateway): dedupe json endpoint prelude 2026-02-15 13:24:37 +00:00
http-utils.ts chore: Enable "experimentalSortImports" in Oxfmt and reformat all imorts. 2026-02-01 10:03:47 +09:00
live-image-probe.ts refactor: consolidate PNG encoder and safeParseJson utilities (#12457) 2026-02-09 00:21:54 -08:00
net.test.ts feat(gateway): add trusted-proxy auth mode (#15940) 2026-02-14 12:32:17 +01:00
net.ts refactor(gateway): share host header parsing 2026-02-15 16:15:53 +00:00
node-command-policy.ts fix(gateway): block node.invoke exec approvals 2026-02-14 19:22:37 +01:00
node-invoke-sanitize.ts refactor(gateway): centralize node.invoke param sanitization 2026-02-14 13:27:45 +01:00
node-invoke-system-run-approval.ts fix(node-host): enforce system.run rawCommand/argv consistency 2026-02-14 18:53:23 +01:00
node-registry.ts chore: Enable "experimentalSortImports" in Oxfmt and reformat all imorts. 2026-02-01 10:03:47 +09:00
open-responses.schema.ts feat(gateway): implement OpenResponses /v1/responses endpoint phase 2 2026-01-20 07:37:01 +00:00
openai-http.e2e.test.ts refactor(channels): dedupe transport and gateway test scaffolds 2026-02-16 14:59:31 +00:00
openai-http.ts refactor(gateway): dedupe assistant delta parsing 2026-02-15 19:08:47 +00:00
openresponses-http.e2e.test.ts refactor(test): dedupe openresponses server setup 2026-02-15 23:34:52 +00:00
openresponses-http.ts refactor(gateway): dedupe assistant delta parsing 2026-02-15 19:08:47 +00:00
openresponses-parity.e2e.test.ts test(gateway): add OpenResponses parity E2E tests 2026-01-20 07:37:01 +00:00
origin-check.test.ts fix: harden control ui framing + ws origin 2026-02-03 16:00:57 -08:00
origin-check.ts refactor(gateway): share host header parsing 2026-02-15 16:15:53 +00:00
probe-auth.ts refactor(gateway): dedupe probe auth resolution 2026-02-15 06:40:04 +00:00
probe.ts refactor: consolidate duplicate utility functions (#12439) 2026-02-08 23:59:43 -08:00
server-broadcast.ts refactor(gateway): share broadcast function types 2026-02-15 13:39:59 +00:00
server-browser.ts chore: Enable "curly" rule to avoid single-statement if confusion/errors. 2026-01-31 16:19:20 +09:00
server-channels.ts gateway: expose isManuallyStopped and resetRestartAttempts on ChannelManager 2026-02-16 23:53:35 +01:00
server-chat.agent-events.test.ts refactor(test): share server chat event harness 2026-02-15 15:44:14 +00:00
server-chat.ts Gateway: bound agent run sequence tracking 2026-02-14 17:50:49 -08:00
server-close.ts chore: Enable "experimentalSortImports" in Oxfmt and reformat all imorts. 2026-02-01 10:03:47 +09:00
server-constants.ts fix(canvas): port remaining iOS branch stability fixes (#18228) 2026-02-16 16:42:28 +00:00
server-cron.ts cron: log model+token usage per run + add usage report script 2026-02-16 23:58:38 +01:00
server-discovery-runtime.ts refactor: rename to openclaw 2026-01-30 03:16:21 +01:00
server-discovery.test.ts refactor: rename to openclaw 2026-01-30 03:16:21 +01:00
server-discovery.ts chore: Enable "curly" rule to avoid single-statement if confusion/errors. 2026-01-31 16:19:20 +09:00
server-http.hooks-request-timeout.test.ts fix(gateway): prune expired entries instead of clearing all hook auth failure state (#15848) 2026-02-14 01:46:12 +01:00
server-http.ts fix(gateway): prune expired entries instead of clearing all hook auth failure state (#15848) 2026-02-14 01:46:12 +01:00
server-lanes.ts refactor: use command lane enum 2026-01-20 10:51:25 +00:00
server-maintenance.ts Gateway: bound agent run sequence tracking 2026-02-14 17:50:49 -08:00
server-methods-list.ts Add mesh auto-planning with chat command UX and hardened auth/session behavior 2026-02-16 23:58:23 +01:00
server-methods.ts Add mesh auto-planning with chat command UX and hardened auth/session behavior 2026-02-16 23:58:23 +01:00
server-mobile-nodes.ts chore: Enable "curly" rule to avoid single-statement if confusion/errors. 2026-01-31 16:19:20 +09:00
server-model-catalog.ts refactor(gateway): split server runtime 2026-01-14 09:11:21 +00:00
server-node-events-types.ts refactor: remove bridge protocol 2026-01-19 10:08:29 +00:00
server-node-events.test.ts agents: reduce prompt token bloat from exec and context (#16539) 2026-02-14 18:32:45 -05:00
server-node-events.ts refactor(gateway): share node event sessionKey parsing 2026-02-15 18:02:55 +00:00
server-node-subscriptions.ts chore: Enable "curly" rule to avoid single-statement if confusion/errors. 2026-01-31 16:19:20 +09:00
server-plugins.test.ts refactor: rename to openclaw 2026-01-30 03:16:21 +01:00
server-plugins.ts chore: Enable "experimentalSortImports" in Oxfmt and reformat all imorts. 2026-02-01 10:03:47 +09:00
server-reload-handlers.ts refactor: unify gateway restart deferral and dispatcher cleanup 2026-02-14 00:38:18 +01:00
server-restart-deferral.test.ts perf(test): cut gateway unit suite overhead 2026-02-16 00:18:26 +00:00
server-restart-sentinel.ts fix: allow agent workspace directories in media local roots (#17136) 2026-02-15 10:53:45 -05:00
server-runtime-config.test.ts feat(gateway): add trusted-proxy auth mode (#15940) 2026-02-14 12:32:17 +01:00
server-runtime-config.ts feat(gateway): add trusted-proxy auth mode (#15940) 2026-02-14 12:32:17 +01:00
server-runtime-state.ts refactor(gateway): share broadcast function types 2026-02-15 13:39:59 +00:00
server-session-key.ts chore: Enable "curly" rule to avoid single-statement if confusion/errors. 2026-01-31 16:19:20 +09:00
server-shared.ts
…
server-startup-log.ts chore: Enable "experimentalSortImports" in Oxfmt and reformat all imorts. 2026-02-01 10:03:47 +09:00
server-startup-memory.test.ts fix (gateway/memory): start qmd onBoot for all agents 2026-02-16 10:35:26 -08:00
server-startup-memory.ts fix (gateway/memory): start qmd onBoot for all agents 2026-02-16 10:35:26 -08:00
server-startup.ts fix: release stale session locks and add watchdog for hung API calls (#18060) 2026-02-16 23:59:22 +01:00
server-tailscale.ts chore: migrate to oxlint and oxfmt 2026-01-14 15:02:19 +00:00
server-utils.ts chore: Enable "curly" rule to avoid single-statement if confusion/errors. 2026-01-31 16:19:20 +09:00
server-wizard-sessions.ts chore: Enable "curly" rule to avoid single-statement if confusion/errors. 2026-01-31 16:19:20 +09:00
server-ws-runtime.ts feat(gateway): add auth rate-limiting & brute-force protection (#15035) 2026-02-13 15:32:38 +01:00
server.agent.gateway-server-agent-a.e2e.test.ts refactor(test): dedupe agent and memory cli test setup 2026-02-16 17:57:45 +00:00
server.agent.gateway-server-agent-b.e2e.test.ts refactor(channels): dedupe transport and gateway test scaffolds 2026-02-16 14:59:31 +00:00
server.agent.gateway-server-agent.mocks.ts fix(test): complete gateway plugin registry mock 2026-02-14 23:51:41 +00:00
server.auth.e2e.test.ts refactor(channels): dedupe transport and gateway test scaffolds 2026-02-16 14:59:31 +00:00
server.canvas-auth.e2e.test.ts refactor(channels): dedupe transport and gateway test scaffolds 2026-02-16 14:59:31 +00:00
server.channels.e2e.test.ts refactor(test): share gateway e2e registry helper 2026-02-15 21:41:18 +00:00
server.chat.gateway-server-chat-b.e2e.test.ts gateway: hard-cap chat.history oversized payloads 2026-02-16 23:56:05 +01:00
server.chat.gateway-server-chat.e2e.test.ts refactor(channels): dedupe transport and gateway test scaffolds 2026-02-16 14:59:31 +00:00
server.config-apply.e2e.test.ts perf(test): stabilize e2e harness and reduce flaky gateway coverage 2026-02-13 17:32:14 +00:00
server.config-patch.e2e.test.ts test(gateway): cover mixed-id config.patch rollback 2026-02-16 08:24:55 -05:00
server.cron.e2e.test.ts cron: separate webhook POST delivery from announce (#17901) 2026-02-16 02:36:00 -08:00
server.e2e-registry-helpers.ts refactor(channels): dedupe transport and gateway test scaffolds 2026-02-16 14:59:31 +00:00
server.e2e-ws-harness.ts refactor(test): reuse env snapshot in gateway ws harness 2026-02-15 23:56:57 +00:00
server.health.e2e.test.ts refactor(test): share gateway ws e2e harness 2026-02-15 22:19:08 +00:00
server.hooks.e2e.test.ts refactor(test): share gateway server start helper 2026-02-15 23:02:27 +00:00
server.impl.ts gateway: wire channel health monitor into startup with configurable interval 2026-02-16 23:53:35 +01:00
server.ios-client-id.e2e.test.ts refactor(test): dedupe ios/android gateway client id tests 2026-02-15 23:07:50 +00:00
server.models-voicewake-misc.e2e.test.ts refactor(test): simplify voicewake env cleanup 2026-02-15 23:34:02 +00:00
server.node-invoke-approval-bypass.e2e.test.ts refactor(channels): dedupe transport and gateway test scaffolds 2026-02-16 14:59:31 +00:00
server.plugin-http-auth.test.ts refactor(channels): dedupe transport and gateway test scaffolds 2026-02-16 14:59:31 +00:00
server.reload.e2e.test.ts refactor(test): share gateway server start helper 2026-02-15 23:02:27 +00:00
server.roles-allowlist-update.e2e.test.ts refactor(channels): dedupe transport and gateway test scaffolds 2026-02-16 14:59:31 +00:00
server.sessions-send.e2e.test.ts fix(security): scope session tools and webhook secret fallback 2026-02-16 03:47:10 +01:00
server.sessions.gateway-server-sessions-a.e2e.test.ts refactor(channels): dedupe transport and gateway test scaffolds 2026-02-16 14:59:31 +00:00
server.skills-status.e2e.test.ts refactor(channels): dedupe transport and gateway test scaffolds 2026-02-16 14:59:31 +00:00
server.talk-config.e2e.test.ts refactor(channels): dedupe transport and gateway test scaffolds 2026-02-16 14:59:31 +00:00
server.ts chore: migrate to oxlint and oxfmt 2026-01-14 15:02:19 +00:00
session-utils.fs.test.ts refactor(channels): dedupe transport and gateway test scaffolds 2026-02-16 14:59:31 +00:00
session-utils.fs.ts refactor(channels): dedupe transport and gateway test scaffolds 2026-02-16 14:59:31 +00:00
session-utils.test.ts fix(sessions): resolve transcript paths with explicit agent context (#16288) 2026-02-14 13:44:51 -05:00
session-utils.ts perf(gateway): optimize sessions/ws/routing 2026-02-14 12:21:44 +00:00
session-utils.types.ts fix: /status shows incorrect context percentage — totalTokens clamped to contextTokens (#15114) (#15133) 2026-02-12 23:52:19 -05:00
sessions-patch.test.ts Agents: add nested subagent orchestration controls and reduce subagent token waste (#14447) 2026-02-14 22:03:45 -08:00
sessions-patch.ts Agents: add nested subagent orchestration controls and reduce subagent token waste (#14447) 2026-02-14 22:03:45 -08:00
sessions-resolve.ts fix(gateway): normalize session key casing to prevent ghost sessions (#12846) 2026-02-13 20:42:24 +01:00
test-helpers.e2e.ts refactor(channels): dedupe transport and gateway test scaffolds 2026-02-16 14:59:31 +00:00
test-helpers.mocks.ts security: redact credentials from config.get gateway responses (#9858) 2026-02-05 16:34:48 -08:00
test-helpers.openai-mock.ts chore: Enable "curly" rule to avoid single-statement if confusion/errors. 2026-01-31 16:19:20 +09:00
test-helpers.server.ts refactor(test): dedupe gateway env restores 2026-02-15 23:18:16 +00:00
test-helpers.ts refactor(src): split oversized modules 2026-01-14 01:17:56 +00:00
test-http-response.ts refactor(channels): dedupe transport and gateway test scaffolds 2026-02-16 14:59:31 +00:00
test-openai-responses-model.ts refactor(channels): dedupe transport and gateway test scaffolds 2026-02-16 14:59:31 +00:00
test-temp-config.ts refactor(channels): dedupe transport and gateway test scaffolds 2026-02-16 14:59:31 +00:00
test-with-server.ts refactor(channels): dedupe transport and gateway test scaffolds 2026-02-16 14:59:31 +00:00
tools-invoke-http.test.ts perf(test): mock config stack in tools invoke http tests 2026-02-14 18:46:24 +00:00
tools-invoke-http.ts refactor(tools): dedupe alsoAllow merge 2026-02-15 16:52:14 +00:00
ws-log.test.ts fix: add agent context to ws logs 2026-01-17 20:37:36 +00:00
ws-log.ts refactor(channels): dedupe transport and gateway test scaffolds 2026-02-16 14:59:31 +00:00
ws-logging.ts
…