diff --git a/CHANGELOG.md b/CHANGELOG.md index edf848fc24f..1532f442531 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -110,6 +110,34 @@ Docs: https://docs.openclaw.ai - Config/All Settings: keep the raw config view intact when sensitive fields are blank instead of corrupting or dropping the rendered snapshot. (#28214) Thanks @solodmd. - Plugin SDK/facades: back-fill bundled plugin facade sentinels before plugin-id tracking re-enters config loading, so CLI/provider startup no longer crashes with `shouldNormalizeGoogleProviderConfig is not a function` or other empty-facade reads during bundled plugin re-entry. Thanks @adam91holt. - Plugins/facades: back-fill facade sentinels before tracked-plugin resolution re-enters config loading, so facade exports stay defined during circular provider normalization. (#61180) Thanks @adam91holt. +- Discord/image generation: include the real generated `MEDIA:` paths in tool output and avoid duplicate plain-output media requeueing so Discord image replies stop pointing at missing local files. +- Slack: route live DM replies back to the concrete inbound DM channel while keeping persisted routing metadata user-scoped, so normal assistant replies stop disappearing when pairing and system messages still arrive. (#59030) Thanks @afurm. +- Discord/reply tags: strip leaked `[[reply_to_current]]` control tags from preview text and honor explicit reply-tag threading during final delivery, so Discord replies stay attached to the triggering message instead of printing reply metadata into chat. +- Telegram: fix current-model checks in the model picker, HTML-format non-default `/model` confirmations, explicit topic replies, persisted reaction ownership across restarts, caption-media placeholder and `file_id` preservation on download failure, and upgraded-install inbound image reads. (#60384, #60042, #59634, #59207, #59948, #59971) Thanks @sfuminya, @GitZhangChi, @dashhuang, @samzong, @v1p0r, and @neeravmakwana. +- Telegram: restore DM voice-note preflight transcription so direct-message audio stops arriving as raw `` placeholders. (#61008) Thanks @manueltarouca. +- Telegram/reasoning: only create a Telegram reasoning preview lane when the session is explicitly `reasoning:stream`, so hidden `` traces from streamed replies stop surfacing as chat previews on normal sessions. Thanks @vincentkoc. +- Telegram/native command menu: trim long menu descriptions before dropping commands so sub-100 command sets can still fit Telegram's payload budget and keep more `/` entries visible. (#61129) Thanks @neeravmakwana. +- Feishu/reasoning: only expose streamed reasoning previews when the session is explicitly `reasoning:stream`, so hidden reasoning traces do not surface on normal streaming sessions. Thanks @vincentkoc. +- Discord: keep REST, webhook, and monitor traffic on the configured proxy, preserve component-only media sends, honor `@everyone` and `@here` mention gates, keep ACK reactions on the active account, and split voice connect/playback timeouts so auto-join is more reliable. (#57465, #60361, #60345) Thanks @geekhuashan. +- WhatsApp: restore `channels.whatsapp.blockStreaming` and reset watchdog timeouts after reconnect so quiet chats stop falling into reconnect loops. (#60007, #60069) Thanks @MonkeyLeeT and @mcaxtr. +- Memory: keep `memory-core` builtin embedding registration on the already-registered path so selecting `memory-core` no longer recurses through plugin discovery and crashes during startup. (#61402) Thanks @ngutman. +- MS Teams: download inline DM images via Graph API and preserve channel reply threading in proactive fallback. (#52212, #55198) Thanks @Ted-developer and @hyojin. +- MS Teams: replace the deprecated Teams SDK HttpPlugin stub with `httpServerAdapter` so recurring gateway deprecation warnings stop firing and the Express 5 compatibility workaround stays on the supported SDK path. (#60939) Thanks @coolramukaka-sys. +- Matrix/exec approvals: anchor seeded approval reactions to the primary Matrix prompt event, resolve them from event metadata instead of prompt text, and clean up chunked approval prompts correctly. (#60931) Thanks @gumadeiras. +- Matrix: recover more reliably when secret storage or recovery keys are missing by recreating secret storage during repair and backup reset, hold crypto snapshot locks during persistence, and surface explicit too-large attachment markers. (#59846, #59851, #60599, #60289) Thanks @al3mart, @emonty, and @efe-arv. +- Android/Talk Mode: cancel in-flight `talk.speak` playback when speech is explicitly stopped, so stale replies stop starting after barge-in or manual stop. (#61164) Thanks @obviyus. +- Android/Talk Mode: restore spoken assistant replies on node-scoped sessions by keeping reply routing synced to the resolved node session key and pausing mic capture during reply playback. (#60306) Thanks @MKV21. +- Android/Talk Mode: restore voice replies on gateway-backed talk mode sessions by updating embedded runner transport overrides to the current agent transport API. (#61214) Thanks @obviyus. +- Voice-call/OpenAI: pass full plugin config into realtime transcription provider resolution so streaming calls can discover the bundled OpenAI realtime transcription provider again. Fixes #60936. Thanks @sliekens and @vincentkoc. +- Control UI/chat: add a per-session thinking-level picker in the chat header and mobile chat settings, and keep the browser bundle on UI-local thinking/session-key helpers so Safari no longer crashes on Node-only imports before rendering chat controls. +- Control UI: keep Stop visible during tool-only execution, preserve pending-send busy state, and clear stale ClawHub search results as soon as the query changes. (#54528, #59800, #60267) Thanks @chziyue and @frankekn. +- Control UI/avatar: honor `ui.assistant.avatar` when serving `/avatar/:agentId` so Appearance UI avatar paths stop falling back to initials placeholders. (#60778) Thanks @hannasdev. +- Control UI/cron: highlight the Cron refresh button while refresh is in flight so the page's loading state stays visible even when prior data remains on screen. (#60394) Thanks @coder-zhuzm. +- Control UI/Overview: prevent gateway access token/password visibility toggle buttons from overlapping their inputs at narrow widths. (#56924) Thanks @bbddbb1. +- CLI/skills JSON: route `skills list --json`, `skills info --json`, and `skills check --json` output to stdout instead of stderr so machine-readable consumers receive JSON on the expected stream again. (#60914; fixes #57599; landed from contributor PR #57611 by @Aftabbs) Thanks @Aftabbs. +- CLI/Commander: preserve Commander-computed exit codes for argument and help-error paths, and cover the user-argv parse mode in the regression tests so invalid CLI invocations no longer report success when exits are intercepted. (#60923) Thanks @Linux2010. +- Cron: replay interrupted recurring jobs on the first gateway restart instead of waiting for a second restart. (#60583) Thanks @joelnishanth. +- Cron: send failure notifications through the job's primary delivery channel using the same session context as successful delivery when no explicit `failureDestination` is configured. (#60622) Thanks @artwalker. - Live model switching: only treat explicit user-driven model changes as pending live switches, so fallback rotation, heartbeat overrides, and compaction no longer trip `LiveSessionModelSwitchError` before making an API call. (#60266) Thanks @kiranvk-2011. - Exec approvals: reuse durable exact-command `allow-always` approvals in allowlist mode so identical reruns stop prompting, and tighten Windows interpreter/path approval handling so wrapper and malformed-path cases fail closed more consistently. (#59880, #59780, #58040, #59182) Thanks @luoyanglang, @SnowSky1, and @pgondhi987. - Node exec approvals: keep node-host `system.run` approvals bound to the prepared execution plan across async forwarding, so mutable script operands still get approval-time binding and drift revalidation instead of dropping back to unbound execution. @@ -190,6 +218,7 @@ Docs: https://docs.openclaw.ai - Tasks/maintenance: reconcile stale cron and chat-backed CLI task rows against live cron-job and agent-run ownership instead of treating any persisted session key as proof that the task is still running. (#60310) Thanks @lml2468. - Plugins: suppress trust-warning noise during non-activating snapshot and CLI metadata loads. (#61427) Thanks @gumadeiras. - Agents/video generation: accept `agents.defaults.videoGenerationModel` in strict config validation and `openclaw config set/get`, so gateways using `video_generate` no longer fail to boot after enabling a video model. +- Matrix/streaming: add a quiet preview mode for streamed Matrix replies, keep legacy `partial` preview-first behavior, and finalize quiet media captions correctly so previews stop notifying early without dropping final text semantics. (#61450) Thanks @gumadeiras. ## 2026.4.2 diff --git a/docs/channels/matrix.md b/docs/channels/matrix.md index d58c713f423..1fa219bb2da 100644 --- a/docs/channels/matrix.md +++ b/docs/channels/matrix.md @@ -178,9 +178,9 @@ This is a practical baseline config with DM pairing, room allowlist, and E2EE en Matrix reply streaming is opt-in. -Set `channels.matrix.streaming` to `"partial"` when you want OpenClaw to send a single draft reply, -edit that draft in place while the model is generating text, and then finalize it when the reply is -done: +Set `channels.matrix.streaming` to `"partial"` when you want OpenClaw to send a single live preview +reply, edit that preview in place while the model is generating text, and then finalize it when the +reply is done: ```json5 { @@ -193,15 +193,164 @@ done: ``` - `streaming: "off"` is the default. OpenClaw waits for the final reply and sends it once. -- `streaming: "partial"` creates one editable preview message for the current assistant block instead of sending multiple partial messages. -- `blockStreaming: true` enables separate Matrix progress messages. With `streaming: "partial"`, Matrix keeps the live draft for the current block and preserves completed blocks as separate messages. -- When `streaming: "partial"` and `blockStreaming` is off, Matrix only edits the live draft and sends the completed reply once that block or turn finishes. +- `streaming: "partial"` creates one editable preview message for the current assistant block using normal Matrix text messages. This preserves Matrix's legacy preview-first notification behavior, so stock clients may notify on the first streamed preview text instead of the finished block. +- `streaming: "quiet"` creates one editable quiet preview notice for the current assistant block. Use this only when you also configure recipient push rules for finalized preview edits. +- `blockStreaming: true` enables separate Matrix progress messages. With preview streaming enabled, Matrix keeps the live draft for the current block and preserves completed blocks as separate messages. +- When preview streaming is on and `blockStreaming` is off, Matrix edits the live draft in place and finalizes that same event when the block or turn finishes. - If the preview no longer fits in one Matrix event, OpenClaw stops preview streaming and falls back to normal final delivery. - Media replies still send attachments normally. If a stale preview can no longer be reused safely, OpenClaw redacts it before sending the final media reply. - Preview edits cost extra Matrix API calls. Leave streaming off if you want the most conservative rate-limit behavior. `blockStreaming` does not enable draft previews by itself. -Use `streaming: "partial"` for preview edits; then add `blockStreaming: true` only if you also want completed assistant blocks to remain visible as separate progress messages. +Use `streaming: "partial"` or `streaming: "quiet"` for preview edits; then add `blockStreaming: true` only if you also want completed assistant blocks to remain visible as separate progress messages. + +If you need stock Matrix notifications without custom push rules, use `streaming: "partial"` for preview-first behavior or leave `streaming` off for final-only delivery. With `streaming: "off"`: + +- `blockStreaming: true` sends each finished block as a normal notifying Matrix message. +- `blockStreaming: false` sends only the final completed reply as a normal notifying Matrix message. + +### Self-hosted push rules for quiet finalized previews + +If you run your own Matrix infrastructure and want quiet previews to notify only when a block or +final reply is done, set `streaming: "quiet"` and add a per-user push rule for finalized preview edits. + +This is usually a recipient-user setup, not a homeserver-global config change: + +Quick map before you start: + +- recipient user = the person who should receive the notification +- bot user = the OpenClaw Matrix account that sends the reply +- use the recipient user's access token for the API calls below +- match `sender` in the push rule against the bot user's full MXID + +1. Configure OpenClaw to use quiet previews: + +```json5 +{ + channels: { + matrix: { + streaming: "quiet", + }, + }, +} +``` + +2. Make sure the recipient account already receives normal Matrix push notifications. Quiet preview + rules only work if that user already has working pushers/devices. + +3. Get the recipient user's access token. + - Use the receiving user's token, not the bot's token. + - Reusing an existing client session token is usually easiest. + - If you need to mint a fresh token, you can log in through the standard Matrix Client-Server API: + +```bash +curl -sS -X POST \ + "https://matrix.example.org/_matrix/client/v3/login" \ + -H "Content-Type: application/json" \ + --data '{ + "type": "m.login.password", + "identifier": { + "type": "m.id.user", + "user": "@alice:example.org" + }, + "password": "REDACTED" + }' +``` + +4. Verify the recipient account already has pushers: + +```bash +curl -sS \ + -H "Authorization: Bearer $USER_ACCESS_TOKEN" \ + "https://matrix.example.org/_matrix/client/v3/pushers" +``` + +If this returns no active pushers/devices, fix normal Matrix notifications first before adding the +OpenClaw rule below. + +OpenClaw marks finalized text-only preview edits with: + +```json +{ + "com.openclaw.finalized_preview": true +} +``` + +5. Create an override push rule for each recipient account which should receive these notifications: + +```bash +curl -sS -X PUT \ + "https://matrix.example.org/_matrix/client/v3/pushrules/global/override/openclaw-finalized-preview" \ + -H "Authorization: Bearer $USER_ACCESS_TOKEN" \ + -H "Content-Type: application/json" \ + --data '{ + "conditions": [ + { "kind": "event_match", "key": "type", "pattern": "m.room.message" }, + { + "kind": "event_property_is", + "key": "content.m\\.relates_to.rel_type", + "value": "m.replace" + }, + { + "kind": "event_property_is", + "key": "content.com\\.openclaw\\.finalized_preview", + "value": true + }, + { "kind": "event_match", "key": "sender", "pattern": "@bot:example.org" } + ], + "actions": [ + "notify", + { "set_tweak": "sound", "value": "default" }, + { "set_tweak": "highlight", "value": false } + ] + }' +``` + +Replace these values before you run the command: + +- `https://matrix.example.org`: your homeserver base URL +- `$USER_ACCESS_TOKEN`: the receiving user's access token +- `@bot:example.org`: your OpenClaw Matrix bot MXID, not the receiving user's MXID + +The rule is evaluated against the event sender: + +- authenticate with the receiving user's token +- match `sender` against the OpenClaw bot MXID + +6. Verify the rule exists: + +```bash +curl -sS \ + -H "Authorization: Bearer $USER_ACCESS_TOKEN" \ + "https://matrix.example.org/_matrix/client/v3/pushrules/global/override/openclaw-finalized-preview" +``` + +7. Test a streamed reply. In quiet mode, the room should show a quiet draft preview and the final + in-place edit should notify once the block or turn finishes. + +Notes: + +- Create the rule with the receiving user's access token, not the bot's. +- New user-defined `override` rules are inserted ahead of default suppress rules, so no extra ordering parameter is needed. +- This only affects text-only preview edits that OpenClaw can safely finalize in place. Media fallbacks and stale-preview fallbacks still use normal Matrix delivery. +- If `GET /_matrix/client/v3/pushers` shows no pushers, the user does not yet have working Matrix push delivery for this account/device. + +#### Synapse + +For Synapse, the setup above is usually enough by itself: + +- No special `homeserver.yaml` change is required for finalized OpenClaw preview notifications. +- If your Synapse deployment already sends normal Matrix push notifications, the user token + `pushrules` call above is the main setup step. +- If you run Synapse behind a reverse proxy or workers, make sure `/_matrix/client/.../pushrules/` reaches Synapse correctly. +- If you run Synapse workers, make sure pushers are healthy. Push delivery is handled by the main process or `synapse.app.pusher` / configured pusher workers. + +#### Tuwunel + +For Tuwunel, use the same setup flow and push-rule API call shown above: + +- No Tuwunel-specific config is required for the finalized preview marker itself. +- If normal Matrix notifications already work for that user, the user token + `pushrules` call above is the main setup step. +- If notifications seem to disappear while the user is active on another device, check whether `suppress_push_when_active` is enabled. Tuwunel added this option in Tuwunel 1.4.2 on September 12, 2025, and it can intentionally suppress pushes to other devices while one device is active. ## Encryption and verification @@ -833,7 +982,7 @@ Live directory lookup uses the logged-in Matrix account: - `historyLimit`: max room messages to include as group history context. Falls back to `messages.groupChat.historyLimit`. Set `0` to disable. - `replyToMode`: `off`, `first`, or `all`. - `markdown`: optional Markdown rendering configuration for outbound Matrix text. -- `streaming`: `off` (default), `partial`, `true`, or `false`. `partial` and `true` enable single-message draft previews with edit-in-place updates. +- `streaming`: `off` (default), `partial`, `quiet`, `true`, or `false`. `partial` and `true` enable preview-first draft updates with normal Matrix text messages. `quiet` uses non-notifying preview notices for self-hosted push-rule setups. - `blockStreaming`: `true` enables separate progress messages for completed assistant blocks while draft preview streaming is active. - `threadReplies`: `off`, `inbound`, or `always`. - `threadBindings`: per-channel overrides for thread-bound session routing and lifecycle. diff --git a/extensions/matrix/src/config-schema.test.ts b/extensions/matrix/src/config-schema.test.ts index b9c11aa3a02..b820d0e2ad3 100644 --- a/extensions/matrix/src/config-schema.test.ts +++ b/extensions/matrix/src/config-schema.test.ts @@ -78,4 +78,13 @@ describe("MatrixConfigSchema SecretInput", () => { } expect(result.data.rooms?.["!room:example.org"]?.account).toBe("axis"); }); + + it("accepts quiet Matrix streaming mode", () => { + const result = MatrixConfigSchema.safeParse({ + homeserver: "https://matrix.example.org", + accessToken: "token", + streaming: "quiet", + }); + expect(result.success).toBe(true); + }); }); diff --git a/extensions/matrix/src/config-schema.ts b/extensions/matrix/src/config-schema.ts index e2d03713cb2..2bbc360712e 100644 --- a/extensions/matrix/src/config-schema.ts +++ b/extensions/matrix/src/config-schema.ts @@ -84,7 +84,7 @@ export const MatrixConfigSchema = z.object({ groupPolicy: GroupPolicySchema.optional(), contextVisibility: ContextVisibilityModeSchema.optional(), blockStreaming: z.boolean().optional(), - streaming: z.union([z.enum(["partial", "off"]), z.boolean()]).optional(), + streaming: z.union([z.enum(["partial", "quiet", "off"]), z.boolean()]).optional(), replyToMode: z.enum(["off", "first", "all", "batched"]).optional(), threadReplies: z.enum(["off", "inbound", "always"]).optional(), textChunkLimit: z.number().optional(), diff --git a/extensions/matrix/src/matrix/draft-stream.test.ts b/extensions/matrix/src/matrix/draft-stream.test.ts index 8771d03935a..b5b5bbab980 100644 --- a/extensions/matrix/src/matrix/draft-stream.test.ts +++ b/extensions/matrix/src/matrix/draft-stream.test.ts @@ -69,7 +69,7 @@ describe("createMatrixDraftStream", () => { vi.useRealTimers(); }); - it("sends a new message on first update", async () => { + it("sends a normal text preview on first partial update", async () => { const stream = createMatrixDraftStream({ roomId: "!room:test", client, @@ -80,14 +80,36 @@ describe("createMatrixDraftStream", () => { await stream.flush(); expect(sendMessageMock).toHaveBeenCalledTimes(1); + expect(sendMessageMock.mock.calls[0]?.[1]).toMatchObject({ + msgtype: "m.text", + }); expect(stream.eventId()).toBe("$evt1"); }); - it("edits the message on subsequent updates", async () => { + it("sends quiet preview notices when quiet mode is enabled", async () => { const stream = createMatrixDraftStream({ roomId: "!room:test", client, cfg: {} as import("../types.js").CoreConfig, + mode: "quiet", + }); + + stream.update("Hello"); + await stream.flush(); + + expect(sendMessageMock).toHaveBeenCalledTimes(1); + expect(sendMessageMock.mock.calls[0]?.[1]).toMatchObject({ + msgtype: "m.notice", + }); + expect(sendMessageMock.mock.calls[0]?.[1]).not.toHaveProperty("m.mentions"); + }); + + it("edits the message on subsequent quiet updates", async () => { + const stream = createMatrixDraftStream({ + roomId: "!room:test", + client, + cfg: {} as import("../types.js").CoreConfig, + mode: "quiet", }); stream.update("Hello"); @@ -102,13 +124,18 @@ describe("createMatrixDraftStream", () => { // First call = initial send, second call = edit (both go through sendMessage) expect(sendMessageMock).toHaveBeenCalledTimes(2); + expect(sendMessageMock.mock.calls[1]?.[1]).toMatchObject({ + msgtype: "m.notice", + "m.new_content": { msgtype: "m.notice" }, + }); }); - it("coalesces rapid updates within throttle window", async () => { + it("coalesces rapid quiet updates within throttle window", async () => { const stream = createMatrixDraftStream({ roomId: "!room:test", client, cfg: {} as import("../types.js").CoreConfig, + mode: "quiet", }); stream.update("A"); @@ -122,6 +149,11 @@ describe("createMatrixDraftStream", () => { expect(sendMessageMock.mock.calls[0][1]).toMatchObject({ body: "A" }); // Edit uses "* " prefix per Matrix m.replace spec. expect(sendMessageMock.mock.calls[1][1]).toMatchObject({ body: "* ABC" }); + expect(sendMessageMock.mock.calls[0][1]).toMatchObject({ msgtype: "m.notice" }); + expect(sendMessageMock.mock.calls[1][1]).toMatchObject({ + msgtype: "m.notice", + "m.new_content": { msgtype: "m.notice" }, + }); }); it("skips no-op updates", async () => { @@ -178,6 +210,7 @@ describe("createMatrixDraftStream", () => { roomId: "!room:test", client, cfg: {} as import("../types.js").CoreConfig, + mode: "quiet", }); stream.update("Block 1"); @@ -296,7 +329,6 @@ describe("createMatrixDraftStream", () => { expect(sendMessageMock).not.toHaveBeenCalled(); expect(stream.eventId()).toBeUndefined(); - expect(stream.mustDeliverFinalNormally()).toBe(true); expect(log).toHaveBeenCalledWith( expect.stringContaining("preview exceeded single-event limit"), ); @@ -317,7 +349,6 @@ describe("createMatrixDraftStream", () => { await stream.flush(); expect(sendMessageMock).not.toHaveBeenCalled(); - expect(stream.mustDeliverFinalNormally()).toBe(true); expect(log).toHaveBeenCalledWith( expect.stringContaining("preview exceeded single-event limit"), ); diff --git a/extensions/matrix/src/matrix/draft-stream.ts b/extensions/matrix/src/matrix/draft-stream.ts index 851fb902001..c9ca5c9b01c 100644 --- a/extensions/matrix/src/matrix/draft-stream.ts +++ b/extensions/matrix/src/matrix/draft-stream.ts @@ -2,8 +2,25 @@ import { createDraftStreamLoop } from "openclaw/plugin-sdk/channel-lifecycle"; import type { CoreConfig } from "../types.js"; import type { MatrixClient } from "./sdk.js"; import { editMessageMatrix, prepareMatrixSingleText, sendSingleTextMessageMatrix } from "./send.js"; +import { MsgType } from "./send/types.js"; const DEFAULT_THROTTLE_MS = 1000; +type MatrixDraftPreviewMode = "partial" | "quiet"; + +function resolveDraftPreviewOptions(mode: MatrixDraftPreviewMode): { + msgtype: typeof MsgType.Text | typeof MsgType.Notice; + includeMentions?: boolean; +} { + if (mode === "quiet") { + return { + msgtype: MsgType.Notice, + includeMentions: false, + }; + } + return { + msgtype: MsgType.Text, + }; +} export type MatrixDraftStream = { /** Update the draft with the latest accumulated text for the current block. */ @@ -16,8 +33,8 @@ export type MatrixDraftStream = { reset: () => void; /** The event ID of the current draft message, if any. */ eventId: () => string | undefined; - /** The last text successfully sent or edited. */ - lastSentText: () => string; + /** True when the provided text matches the last rendered draft payload. */ + matchesPreparedText: (text: string) => boolean; /** True when preview streaming must fall back to normal final delivery. */ mustDeliverFinalNormally: () => boolean; }; @@ -26,6 +43,7 @@ export function createMatrixDraftStream(params: { roomId: string; client: MatrixClient; cfg: CoreConfig; + mode?: MatrixDraftPreviewMode; threadId?: string; replyToId?: string; /** When true, reset() restores the original replyToId instead of clearing it. */ @@ -34,6 +52,7 @@ export function createMatrixDraftStream(params: { log?: (message: string) => void; }): MatrixDraftStream { const { roomId, client, cfg, threadId, accountId, log } = params; + const preview = resolveDraftPreviewOptions(params.mode ?? "partial"); let currentEventId: string | undefined; let lastSentText = ""; @@ -59,8 +78,6 @@ export function createMatrixDraftStream(params: { ); return false; } - // If the initial send failed, stop trying for this block. The deliver - // callback will fall back to deliverMatrixReplies. if (sendFailed) { return false; } @@ -75,6 +92,8 @@ export function createMatrixDraftStream(params: { replyToId, threadId, accountId, + msgtype: preview.msgtype, + includeMentions: preview.includeMentions, }); currentEventId = result.messageId; lastSentText = preparedText.trimmedText; @@ -85,6 +104,8 @@ export function createMatrixDraftStream(params: { cfg, threadId, accountId, + msgtype: preview.msgtype, + includeMentions: preview.includeMentions, }); lastSentText = preparedText.trimmedText; } @@ -94,16 +115,11 @@ export function createMatrixDraftStream(params: { const isPreviewLimitError = err instanceof Error && err.message.startsWith("Matrix single-message text exceeds limit"); if (isPreviewLimitError) { - // Once the preview no longer fits in one editable event, preserve the - // current preview as-is and fall back to normal final delivery. finalizeInPlaceBlocked = true; } if (!currentEventId) { - // First send failed — give up for this block so the deliver callback - // falls through to normal delivery. sendFailed = true; } - // Signal failure so the loop stops retrying. stopped = true; return false; } @@ -148,7 +164,11 @@ export function createMatrixDraftStream(params: { stop, reset, eventId: () => currentEventId, - lastSentText: () => lastSentText, + matchesPreparedText: (text: string) => + prepareMatrixSingleText(text, { + cfg, + accountId, + }).trimmedText === lastSentText, mustDeliverFinalNormally: () => sendFailed || finalizeInPlaceBlocked, }; } diff --git a/extensions/matrix/src/matrix/monitor/handler.test-helpers.ts b/extensions/matrix/src/matrix/monitor/handler.test-helpers.ts index 393b64ab682..dbb7614348e 100644 --- a/extensions/matrix/src/matrix/monitor/handler.test-helpers.ts +++ b/extensions/matrix/src/matrix/monitor/handler.test-helpers.ts @@ -1,6 +1,6 @@ import { vi } from "vitest"; import type { RuntimeEnv, RuntimeLogger } from "../../runtime-api.js"; -import type { MatrixRoomConfig, ReplyToMode } from "../../types.js"; +import type { MatrixRoomConfig, MatrixStreamingMode, ReplyToMode } from "../../types.js"; import type { MatrixClient } from "../sdk.js"; import { createMatrixRoomMessageHandler, type MatrixMonitorHandlerParams } from "./handler.js"; import { EventType, type MatrixRawEvent, type RoomMessageEventContent } from "./types.js"; @@ -32,7 +32,7 @@ type MatrixHandlerTestHarnessOptions = { threadReplies?: "off" | "inbound" | "always"; dmThreadReplies?: "off" | "inbound" | "always"; dmSessionScope?: "per-user" | "per-room"; - streaming?: "partial" | "off"; + streaming?: MatrixStreamingMode; blockStreamingEnabled?: boolean; dmEnabled?: boolean; dmPolicy?: "pairing" | "allowlist" | "open" | "disabled"; diff --git a/extensions/matrix/src/matrix/monitor/handler.test.ts b/extensions/matrix/src/matrix/monitor/handler.test.ts index 846b67f34de..378bb62b6c0 100644 --- a/extensions/matrix/src/matrix/monitor/handler.test.ts +++ b/extensions/matrix/src/matrix/monitor/handler.test.ts @@ -8,6 +8,7 @@ import { } from "openclaw/plugin-sdk/conversation-runtime"; import { beforeEach, describe, expect, it, vi } from "vitest"; import { installMatrixMonitorTestRuntime } from "../../test-runtime.js"; +import { MATRIX_OPENCLAW_FINALIZED_PREVIEW_KEY } from "../send/types.js"; import { createMatrixRoomMessageHandler } from "./handler.js"; import { createMatrixHandlerTestHarness, @@ -1988,6 +1989,7 @@ describe("matrix monitor handler draft streaming", () => { function createStreamingHarness(opts?: { replyToMode?: "off" | "first" | "all" | "batched"; blockStreamingEnabled?: boolean; + streaming?: "partial" | "quiet"; }) { let capturedDeliver: DeliverFn | undefined; let capturedReplyOpts: ReplyOpts | undefined; @@ -2007,7 +2009,7 @@ describe("matrix monitor handler draft streaming", () => { const redactEventMock = vi.fn(async () => "$redacted"); const { handler } = createMatrixHandlerTestHarness({ - streaming: "partial", + streaming: opts?.streaming ?? "quiet", blockStreamingEnabled: opts?.blockStreamingEnabled ?? false, replyToMode: opts?.replyToMode ?? "off", client: { redactEvent: redactEventMock }, @@ -2067,7 +2069,7 @@ describe("matrix monitor handler draft streaming", () => { return { dispatch, redactEventMock }; } - it("finalizes a single partial-preview block in place when block streaming is enabled", async () => { + it("finalizes a single quiet-preview block in place when block streaming is enabled", async () => { const { dispatch, redactEventMock } = createStreamingHarness({ blockStreamingEnabled: true }); const { deliver, opts, finish } = await dispatch(); @@ -2080,12 +2082,76 @@ describe("matrix monitor handler draft streaming", () => { await deliver({ text: "Single block" }, { kind: "final" }); expect(sendSingleTextMessageMatrixMock).toHaveBeenCalledTimes(1); + expect(editMessageMatrixMock).toHaveBeenCalledWith( + "!room:example.org", + "$draft1", + "Single block", + expect.objectContaining({ + extraContent: { [MATRIX_OPENCLAW_FINALIZED_PREVIEW_KEY]: true }, + }), + ); expect(deliverMatrixRepliesMock).not.toHaveBeenCalled(); expect(redactEventMock).not.toHaveBeenCalled(); await finish(); }); - it("preserves completed blocks by rotating to a new draft when block streaming is enabled", async () => { + it("keeps partial preview-first finalization on the existing draft when text is unchanged", async () => { + const { dispatch, redactEventMock } = createStreamingHarness({ + blockStreamingEnabled: true, + streaming: "partial", + }); + const { deliver, opts, finish } = await dispatch(); + + opts.onPartialReply?.({ text: "Single block" }); + await vi.waitFor(() => { + expect(sendSingleTextMessageMatrixMock).toHaveBeenCalledTimes(1); + }); + + expect(sendSingleTextMessageMatrixMock).toHaveBeenCalledWith( + "!room:example.org", + "Single block", + expect.not.objectContaining({ + msgtype: "m.notice", + includeMentions: false, + }), + ); + + await deliver({ text: "Single block" }, { kind: "final" }); + + expect(editMessageMatrixMock).not.toHaveBeenCalled(); + expect(deliverMatrixRepliesMock).not.toHaveBeenCalled(); + expect(redactEventMock).not.toHaveBeenCalled(); + await finish(); + }); + + it("still edits partial preview-first drafts when the final text changes", async () => { + const { dispatch, redactEventMock } = createStreamingHarness({ + blockStreamingEnabled: true, + streaming: "partial", + }); + const { deliver, opts, finish } = await dispatch(); + + opts.onPartialReply?.({ text: "Single" }); + await vi.waitFor(() => { + expect(sendSingleTextMessageMatrixMock).toHaveBeenCalledTimes(1); + }); + + await deliver({ text: "Single block" }, { kind: "final" }); + + expect(editMessageMatrixMock).toHaveBeenCalledWith( + "!room:example.org", + "$draft1", + "Single block", + expect.not.objectContaining({ + extraContent: { [MATRIX_OPENCLAW_FINALIZED_PREVIEW_KEY]: true }, + }), + ); + expect(deliverMatrixRepliesMock).not.toHaveBeenCalled(); + expect(redactEventMock).not.toHaveBeenCalled(); + await finish(); + }); + + it("preserves completed blocks by rotating to a new quiet preview", async () => { const { dispatch, redactEventMock } = createStreamingHarness({ blockStreamingEnabled: true }); const { deliver, opts, finish } = await dispatch(); @@ -2097,6 +2163,14 @@ describe("matrix monitor handler draft streaming", () => { deliverMatrixRepliesMock.mockClear(); await deliver({ text: "Block one" }, { kind: "block" }); + expect(editMessageMatrixMock).toHaveBeenCalledWith( + "!room:example.org", + "$draft1", + "Block one", + expect.objectContaining({ + extraContent: { [MATRIX_OPENCLAW_FINALIZED_PREVIEW_KEY]: true }, + }), + ); expect(deliverMatrixRepliesMock).not.toHaveBeenCalled(); expect(redactEventMock).not.toHaveBeenCalled(); @@ -2112,6 +2186,14 @@ describe("matrix monitor handler draft streaming", () => { await deliver({ text: "Block two" }, { kind: "final" }); + expect(editMessageMatrixMock).toHaveBeenCalledWith( + "!room:example.org", + "$draft2", + "Block two", + expect.objectContaining({ + extraContent: { [MATRIX_OPENCLAW_FINALIZED_PREVIEW_KEY]: true }, + }), + ); expect(deliverMatrixRepliesMock).not.toHaveBeenCalled(); expect(redactEventMock).not.toHaveBeenCalled(); await finish(); @@ -2198,17 +2280,13 @@ describe("matrix monitor handler draft streaming", () => { const { dispatch } = createStreamingHarness(); const { deliver, opts, finish } = await dispatch(); - // Simulate streaming: partial reply creates draft message. opts.onPartialReply?.({ text: "Hello" }); - // Wait for the draft stream's immediate send to complete. await vi.waitFor(() => { expect(sendSingleTextMessageMatrixMock).toHaveBeenCalledTimes(1); }); - // Make the final edit fail. editMessageMatrixMock.mockRejectedValueOnce(new Error("rate limited")); - // Deliver final — should catch edit failure and fall back. await deliver({ text: "Hello world" }, { kind: "block" }); expect(deliverMatrixRepliesMock).toHaveBeenCalledTimes(1); @@ -2309,6 +2387,8 @@ describe("matrix monitor handler draft streaming", () => { "Alpha", expect.anything(), ); + expect(deliverMatrixRepliesMock).not.toHaveBeenCalled(); + expect(redactEventMock).not.toHaveBeenCalled(); await vi.waitFor(() => { expect(sendSingleTextMessageMatrixMock).toHaveBeenCalledTimes(1); }); @@ -2358,6 +2438,8 @@ describe("matrix monitor handler draft streaming", () => { "Alpha", expect.anything(), ); + expect(deliverMatrixRepliesMock).not.toHaveBeenCalled(); + expect(redactEventMock).not.toHaveBeenCalled(); await vi.waitFor(() => { expect(sendSingleTextMessageMatrixMock).toHaveBeenCalledTimes(1); }); @@ -2400,7 +2482,14 @@ describe("matrix monitor handler draft streaming", () => { expect(sendSingleTextMessageMatrixMock).toHaveBeenCalledTimes(1); }); expect(sendSingleTextMessageMatrixMock.mock.calls[0]?.[1]).toBe("Beta"); - expect(editMessageMatrixMock).not.toHaveBeenCalled(); + expect(editMessageMatrixMock).toHaveBeenCalledWith( + "!room:example.org", + "$draft1", + "Alpha", + expect.objectContaining({ + extraContent: { [MATRIX_OPENCLAW_FINALIZED_PREVIEW_KEY]: true }, + }), + ); sendSingleTextMessageMatrixMock.mockClear(); editMessageMatrixMock.mockClear(); @@ -2414,7 +2503,14 @@ describe("matrix monitor handler draft streaming", () => { expect(sendSingleTextMessageMatrixMock).toHaveBeenCalledTimes(1); }); expect(sendSingleTextMessageMatrixMock.mock.calls[0]?.[1]).toBe("Gamma"); - expect(editMessageMatrixMock).not.toHaveBeenCalled(); + expect(editMessageMatrixMock).toHaveBeenCalledWith( + "!room:example.org", + "$draft2", + "Beta", + expect.objectContaining({ + extraContent: { [MATRIX_OPENCLAW_FINALIZED_PREVIEW_KEY]: true }, + }), + ); await finish(); }); @@ -2431,7 +2527,7 @@ describe("matrix monitor handler draft streaming", () => { let capturedReplyOpts: ReplyOpts | undefined; const { handler } = createMatrixHandlerTestHarness({ - streaming: "partial", + streaming: "quiet", createReplyDispatcherWithTyping: () => ({ dispatcher: { markComplete: () => {}, waitForIdle: async () => {} }, replyOptions: {}, @@ -2555,6 +2651,46 @@ describe("matrix monitor handler draft streaming", () => { await finish(); }); + it("finalizes quiet drafts before reusing unchanged media captions", async () => { + const { dispatch, redactEventMock } = createStreamingHarness({ streaming: "quiet" }); + const { deliver, opts, finish } = await dispatch(); + + opts.onPartialReply?.({ text: "@room screenshot ready" }); + await vi.waitFor(() => { + expect(sendSingleTextMessageMatrixMock).toHaveBeenCalledTimes(1); + }); + + deliverMatrixRepliesMock.mockClear(); + await deliver( + { + text: "@room screenshot ready", + mediaUrl: "https://example.com/image.png", + }, + { kind: "final" }, + ); + + expect(editMessageMatrixMock).toHaveBeenCalledWith( + "!room:example.org", + "$draft1", + "@room screenshot ready", + expect.objectContaining({ + extraContent: { [MATRIX_OPENCLAW_FINALIZED_PREVIEW_KEY]: true }, + }), + ); + expect(redactEventMock).not.toHaveBeenCalled(); + expect(deliverMatrixRepliesMock).toHaveBeenCalledWith( + expect.objectContaining({ + replies: [ + expect.objectContaining({ + mediaUrl: "https://example.com/image.png", + text: undefined, + }), + ], + }), + ); + await finish(); + }); + it("redacts stale draft and sends the final once when a later preview exceeds the event limit", async () => { const { dispatch, redactEventMock } = createStreamingHarness(); const { deliver, opts, finish } = await dispatch(); @@ -2628,6 +2764,27 @@ describe("matrix monitor handler block streaming config", () => { expect(capturedDisableBlockStreaming).toBe(true); }); + it("keeps block streaming disabled when quiet previews are on and block streaming is off", async () => { + let capturedDisableBlockStreaming: boolean | undefined; + + const { handler } = createMatrixHandlerTestHarness({ + streaming: "quiet", + dispatchReplyFromConfig: vi.fn( + async (args: { replyOptions?: { disableBlockStreaming?: boolean } }) => { + capturedDisableBlockStreaming = args.replyOptions?.disableBlockStreaming; + return { queuedFinal: false, counts: { final: 0, block: 0, tool: 0 } }; + }, + ) as never, + }); + + await handler( + "!room:example.org", + createMatrixTextMessageEvent({ eventId: "$msg1", body: "hello" }), + ); + + expect(capturedDisableBlockStreaming).toBe(true); + }); + it("allows shared block streaming when partial previews and block streaming are both enabled", async () => { let capturedDisableBlockStreaming: boolean | undefined; diff --git a/extensions/matrix/src/matrix/monitor/handler.ts b/extensions/matrix/src/matrix/monitor/handler.ts index 190bef919fd..7f8f10bf4db 100644 --- a/extensions/matrix/src/matrix/monitor/handler.ts +++ b/extensions/matrix/src/matrix/monitor/handler.ts @@ -6,7 +6,12 @@ import { } from "openclaw/plugin-sdk/config-runtime"; import { getSessionBindingService } from "openclaw/plugin-sdk/conversation-runtime"; import { evaluateSupplementalContextVisibility } from "openclaw/plugin-sdk/security-runtime"; -import type { CoreConfig, MatrixRoomConfig, ReplyToMode } from "../../types.js"; +import type { + CoreConfig, + MatrixRoomConfig, + MatrixStreamingMode, + ReplyToMode, +} from "../../types.js"; import { createMatrixDraftStream } from "../draft-stream.js"; import { isMatrixMediaSizeLimitError } from "../media-errors.js"; import { @@ -32,6 +37,7 @@ import { sendTypingMatrix, } from "../send.js"; import { resolveMatrixStoredSessionMeta } from "../session-store-metadata.js"; +import { MATRIX_OPENCLAW_FINALIZED_PREVIEW_KEY } from "../send/types.js"; import { resolveMatrixMonitorAccessState } from "./access-state.js"; import { resolveMatrixAckReactionConfig } from "./ack-config.js"; import { resolveMatrixAllowListMatch } from "./allowlist.js"; @@ -76,6 +82,18 @@ const MAX_TRACKED_PAIRING_REPLY_SENDERS = 512; const MAX_TRACKED_SHARED_DM_CONTEXT_NOTICES = 512; type MatrixAllowBotsMode = "off" | "mentions" | "all"; +async function redactMatrixDraftEvent( + client: MatrixClient, + roomId: string, + draftEventId: string, +): Promise { + await client.redactEvent(roomId, draftEventId).catch(() => {}); +} + +function buildMatrixFinalizedPreviewContent(): Record { + return { [MATRIX_OPENCLAW_FINALIZED_PREVIEW_KEY]: true }; +} + export type MatrixMonitorHandlerParams = { client: MatrixClient; core: PluginRuntime; @@ -96,7 +114,7 @@ export type MatrixMonitorHandlerParams = { dmThreadReplies?: "off" | "inbound" | "always"; /** DM session grouping behavior. */ dmSessionScope?: "per-user" | "per-room"; - streaming: "partial" | "off"; + streaming: MatrixStreamingMode; blockStreamingEnabled: boolean; dmEnabled: boolean; dmPolicy: "open" | "pairing" | "allowlist" | "disabled"; @@ -1280,14 +1298,15 @@ export function createMatrixRoomMessageHandler(params: MatrixMonitorHandlerParam }); }, }); - const draftStreamingEnabled = streaming === "partial"; + const draftStreamingEnabled = streaming !== "off"; + const quietDraftStreaming = streaming === "quiet"; const draftReplyToId = replyToMode !== "off" && !threadTarget ? _messageId : undefined; - let currentDraftReplyToId = draftReplyToId; const draftStream = draftStreamingEnabled ? createMatrixDraftStream({ roomId, client, cfg, + mode: quietDraftStreaming ? "quiet" : "partial", threadId: threadTarget, replyToId: draftReplyToId, preserveReplyId: replyToMode === "all", @@ -1308,6 +1327,7 @@ export function createMatrixRoomMessageHandler(params: MatrixMonitorHandlerParam let latestDraftFullText = ""; const pendingDraftBoundaries: PendingDraftBoundary[] = []; const latestQueuedDraftBoundaryOffsets = new Map(); + let currentDraftReplyToId = draftReplyToId; // Set after the first final payload consumes the draft event so // subsequent finals go through normal delivery. let draftConsumed = false; @@ -1380,9 +1400,8 @@ export function createMatrixRoomMessageHandler(params: MatrixMonitorHandlerParam const hasMedia = Boolean(payload.mediaUrl) || (payload.mediaUrls?.length ?? 0) > 0; await draftStream.stop(); + const draftEventId = draftStream.eventId(); - // After the first final payload consumes the draft, subsequent - // finals must go through normal delivery to avoid overwriting. if (draftConsumed) { await deliverMatrixReplies({ cfg, @@ -1400,16 +1419,6 @@ export function createMatrixRoomMessageHandler(params: MatrixMonitorHandlerParam return; } - // Read event id after stop() — flush may have created the - // initial message while draining pending text. - const draftEventId = draftStream.eventId(); - - // If the payload carries a reply target that differs from the - // draft's, fall through to normal delivery — Matrix edits - // cannot change the reply relation on an existing event. - // Skip when replyToMode is "off" (replies stripped anyway) - // or when threadTarget is set (thread relations take - // precedence over replyToId in deliverMatrixReplies). const payloadReplyToId = payload.replyToId?.trim() || undefined; const payloadReplyMismatch = replyToMode !== "off" && @@ -1424,58 +1433,61 @@ export function createMatrixRoomMessageHandler(params: MatrixMonitorHandlerParam !payloadReplyMismatch && !mustDeliverFinalNormally ) { - // Text-only: final edit of the draft message. Skip if - // stop() already flushed identical text to avoid a - // redundant API call that wastes rate-limit budget. - if (payload.text !== draftStream.lastSentText()) { - try { + try { + const requiresFinalEdit = + quietDraftStreaming || !draftStream.matchesPreparedText(payload.text); + if (requiresFinalEdit) { await editMessageMatrix(roomId, draftEventId, payload.text, { client, cfg, threadId: threadTarget, accountId: _route.accountId, - }); - } catch { - // Edit failed (rate limit, server error) — redact the - // stale draft and fall back to normal delivery so the - // user still gets the final answer. - await client.redactEvent(roomId, draftEventId).catch(() => {}); - await deliverMatrixReplies({ - cfg, - replies: [payload], - roomId, - client, - runtime, - textLimit, - replyToMode, - threadId: threadTarget, - accountId: _route.accountId, - mediaLocalRoots, - tableMode, + extraContent: quietDraftStreaming + ? buildMatrixFinalizedPreviewContent() + : undefined, }); } + } catch { + await redactMatrixDraftEvent(client, roomId, draftEventId); + await deliverMatrixReplies({ + cfg, + replies: [payload], + roomId, + client, + runtime, + textLimit, + replyToMode, + threadId: threadTarget, + accountId: _route.accountId, + mediaLocalRoots, + tableMode, + }); } draftConsumed = true; } else if (draftEventId && hasMedia && !payloadReplyMismatch) { - // Media payload: finalize draft text, send media separately. let textEditOk = !mustDeliverFinalNormally; - if (textEditOk && payload.text && payload.text !== draftStream.lastSentText()) { - textEditOk = await editMessageMatrix(roomId, draftEventId, payload.text, { + const payloadText = payload.text; + const requiresFinalTextEdit = + quietDraftStreaming || + (typeof payloadText === "string" && + !draftStream.matchesPreparedText(payloadText)); + if (textEditOk && payloadText && requiresFinalTextEdit) { + textEditOk = await editMessageMatrix(roomId, draftEventId, payloadText, { client, cfg, threadId: threadTarget, accountId: _route.accountId, + extraContent: quietDraftStreaming + ? buildMatrixFinalizedPreviewContent() + : undefined, }).then( () => true, () => false, ); } const reusesDraftAsFinalText = Boolean(payload.text?.trim()) && textEditOk; - // If the text edit failed, or there is no final text to reuse - // the preview, redact the stale draft and include text in media - // delivery so the final caption is not lost. if (!reusesDraftAsFinalText) { - await client.redactEvent(roomId, draftEventId).catch(() => {}); + await redactMatrixDraftEvent(client, roomId, draftEventId); } await deliverMatrixReplies({ cfg, @@ -1494,11 +1506,8 @@ export function createMatrixRoomMessageHandler(params: MatrixMonitorHandlerParam }); draftConsumed = true; } else { - // Redact stale draft when the final delivery will create a - // new message (reply-target mismatch, preview overflow, or no - // usable draft). if (draftEventId && (payloadReplyMismatch || mustDeliverFinalNormally)) { - await client.redactEvent(roomId, draftEventId).catch(() => {}); + await redactMatrixDraftEvent(client, roomId, draftEventId); } await deliverMatrixReplies({ cfg, @@ -1515,9 +1524,6 @@ export function createMatrixRoomMessageHandler(params: MatrixMonitorHandlerParam }); } - // Only reset for intermediate blocks — after the final delivery - // the stream must stay stopped so late async callbacks cannot - // create ghost messages. if (info.kind === "block") { draftConsumed = false; advanceDraftBlockBoundary({ fallbackToLatestEnd: true }); diff --git a/extensions/matrix/src/matrix/monitor/index.ts b/extensions/matrix/src/matrix/monitor/index.ts index 5c70504c2ce..7b301084d53 100644 --- a/extensions/matrix/src/matrix/monitor/index.ts +++ b/extensions/matrix/src/matrix/monitor/index.ts @@ -215,8 +215,12 @@ export async function monitorMatrixProvider(opts: MonitorMatrixOpts = {}): Promi const historyLimit = Math.max(0, accountConfig.historyLimit ?? globalGroupChatHistoryLimit ?? 0); const mediaMaxMb = opts.mediaMaxMb ?? accountConfig.mediaMaxMb ?? DEFAULT_MEDIA_MAX_MB; const mediaMaxBytes = Math.max(1, mediaMaxMb) * 1024 * 1024; - const streaming: "partial" | "off" = - accountConfig.streaming === true || accountConfig.streaming === "partial" ? "partial" : "off"; + const streaming: "partial" | "quiet" | "off" = + accountConfig.streaming === true || accountConfig.streaming === "partial" + ? "partial" + : accountConfig.streaming === "quiet" + ? "quiet" + : "off"; const blockStreamingEnabled = accountConfig.blockStreaming === true; const startupMs = Date.now(); const startupGraceMs = 0; diff --git a/extensions/matrix/src/matrix/send.test.ts b/extensions/matrix/src/matrix/send.test.ts index 1275b7766cd..9ff9da0c364 100644 --- a/extensions/matrix/src/matrix/send.test.ts +++ b/extensions/matrix/src/matrix/send.test.ts @@ -9,6 +9,7 @@ import { sendSingleTextMessageMatrix, sendTypingMatrix, } from "./send.js"; +import { MATRIX_OPENCLAW_FINALIZED_PREVIEW_KEY } from "./send/types.js"; const loadOutboundMediaFromUrlMock = vi.hoisted(() => vi.fn()); const loadWebMediaMock = vi.fn().mockResolvedValue({ @@ -602,6 +603,39 @@ describe("sendSingleTextMessageMatrix", () => { expect(sendMessage).not.toHaveBeenCalled(); }); + + it("supports quiet draft preview sends without mention metadata", async () => { + const { client, sendMessage } = makeClient(); + + await sendSingleTextMessageMatrix("room:!room:example", "@room hi @alice:example.org", { + client, + msgtype: "m.notice", + includeMentions: false, + }); + + expect(sendMessage.mock.calls[0]?.[1]).toMatchObject({ + msgtype: "m.notice", + body: "@room hi @alice:example.org", + }); + expect(sendMessage.mock.calls[0]?.[1]).not.toHaveProperty("m.mentions"); + expect( + (sendMessage.mock.calls[0]?.[1] as { formatted_body?: string }).formatted_body, + ).not.toContain("matrix.to"); + }); + + it("merges extra content fields into single-event sends", async () => { + const { client, sendMessage } = makeClient(); + + await sendSingleTextMessageMatrix("room:!room:example", "done", { + client, + extraContent: { [MATRIX_OPENCLAW_FINALIZED_PREVIEW_KEY]: true }, + }); + + expect(sendMessage.mock.calls[0]?.[1]).toMatchObject({ + body: "done", + [MATRIX_OPENCLAW_FINALIZED_PREVIEW_KEY]: true, + }); + }); }); describe("editMessageMatrix mentions", () => { @@ -677,6 +711,57 @@ describe("editMessageMatrix mentions", () => { }, }); }); + + it("supports quiet draft preview edits without mention metadata", async () => { + const { client, sendMessage, getEvent } = makeClient(); + getEvent.mockResolvedValue({ + content: { + body: "@room hi @alice:example.org", + "m.mentions": { room: true, user_ids: ["@alice:example.org"] }, + }, + }); + + await editMessageMatrix("room:!room:example", "$original", "@room hi @alice:example.org", { + client, + msgtype: "m.notice", + includeMentions: false, + }); + + expect(sendMessage.mock.calls[0]?.[1]).toMatchObject({ + msgtype: "m.notice", + "m.new_content": { + msgtype: "m.notice", + }, + }); + expect(sendMessage.mock.calls[0]?.[1]).not.toHaveProperty("m.mentions"); + expect(sendMessage.mock.calls[0]?.[1]?.["m.new_content"]).not.toHaveProperty("m.mentions"); + expect( + (sendMessage.mock.calls[0]?.[1] as { formatted_body?: string }).formatted_body, + ).not.toContain("matrix.to"); + expect( + ( + sendMessage.mock.calls[0]?.[1] as { + "m.new_content"?: { formatted_body?: string }; + } + )["m.new_content"]?.formatted_body, + ).not.toContain("matrix.to"); + }); + + it("merges extra content fields into edit payloads and m.new_content", async () => { + const { client, sendMessage } = makeClient(); + + await editMessageMatrix("room:!room:example", "$original", "done", { + client, + extraContent: { [MATRIX_OPENCLAW_FINALIZED_PREVIEW_KEY]: true }, + }); + + expect(sendMessage.mock.calls[0]?.[1]).toMatchObject({ + [MATRIX_OPENCLAW_FINALIZED_PREVIEW_KEY]: true, + "m.new_content": { + [MATRIX_OPENCLAW_FINALIZED_PREVIEW_KEY]: true, + }, + }); + }); }); describe("sendPollMatrix mentions", () => { diff --git a/extensions/matrix/src/matrix/send.ts b/extensions/matrix/src/matrix/send.ts index 4fddfde2806..7c40e9b41cc 100644 --- a/extensions/matrix/src/matrix/send.ts +++ b/extensions/matrix/src/matrix/send.ts @@ -33,9 +33,11 @@ import { EventType, MsgType, RelationType, + type MatrixExtraContentFields, type MatrixOutboundContent, type MatrixSendOpts, type MatrixSendResult, + type MatrixTextMsgType, } from "./send/types.js"; const MATRIX_TEXT_LIMIT = 4000; @@ -102,6 +104,16 @@ function hasMatrixMentionsMetadata(content: Record | undefined) return Boolean(content && Object.hasOwn(content, "m.mentions")); } +function withMatrixExtraContentFields>( + content: T, + extraContent?: MatrixExtraContentFields, +): T { + if (!extraContent) { + return content; + } + return { ...content, ...extraContent }; +} + async function resolvePreviousEditMentions(params: { client: MatrixClient; content: Record | undefined; @@ -398,6 +410,9 @@ export async function sendSingleTextMessageMatrix( replyToId?: string; threadId?: string; accountId?: string; + msgtype?: MatrixTextMsgType; + includeMentions?: boolean; + extraContent?: MatrixExtraContentFields; } = {}, ): Promise { const { trimmedText, convertedText, singleEventLimit, fitsInSingleEvent } = @@ -425,11 +440,17 @@ export async function sendSingleTextMessageMatrix( const relation = normalizedThreadId ? buildThreadRelation(normalizedThreadId, opts.replyToId) : buildReplyRelation(opts.replyToId); - const content = buildTextContent(convertedText, relation); + const content = withMatrixExtraContentFields( + buildTextContent(convertedText, relation, { + msgtype: opts.msgtype, + }), + opts.extraContent, + ); await enrichMatrixFormattedContent({ client, content, markdown: convertedText, + includeMentions: opts.includeMentions, }); const eventId = await client.sendMessage(resolvedRoom, content); return { @@ -468,6 +489,9 @@ export async function editMessageMatrix( threadId?: string; accountId?: string; timeoutMs?: number; + msgtype?: MatrixTextMsgType; + includeMentions?: boolean; + extraContent?: MatrixExtraContentFields; } = {}, ): Promise { return await withResolvedMatrixSendClient( @@ -486,22 +510,30 @@ export async function editMessageMatrix( accountId: opts.accountId, }); const convertedText = getCore().channel.text.convertMarkdownTables(newText, tableMode); - const newContent = buildTextContent(convertedText); + const newContent = withMatrixExtraContentFields( + buildTextContent(convertedText, undefined, { + msgtype: opts.msgtype, + }), + opts.extraContent, + ); await enrichMatrixFormattedContent({ client, content: newContent, markdown: convertedText, + includeMentions: opts.includeMentions, }); - const previousEvent = await getPreviousMatrixEvent(client, resolvedRoom, originalEventId); - const previousContent = resolvePreviousEditContent(previousEvent); - const previousMentions = await resolvePreviousEditMentions({ - client, - content: previousContent, - }); - const replaceMentions = diffMatrixMentions( - extractMatrixMentions(newContent), - previousMentions, - ); + const replaceMentions = + opts.includeMentions === false + ? undefined + : diffMatrixMentions( + extractMatrixMentions(newContent), + await resolvePreviousEditMentions({ + client, + content: resolvePreviousEditContent( + await getPreviousMatrixEvent(client, resolvedRoom, originalEventId), + ), + }), + ); const replaceRelation: Record = { rel_type: RelationType.Replace, @@ -522,10 +554,12 @@ export async function editMessageMatrix( ...(typeof newContent.formatted_body === "string" ? { formatted_body: `* ${newContent.formatted_body}` } : {}), - "m.mentions": replaceMentions, "m.new_content": newContent, "m.relates_to": replaceRelation, }; + if (replaceMentions !== undefined) { + content["m.mentions"] = replaceMentions; + } const eventId = await client.sendMessage(resolvedRoom, content); return eventId ?? ""; diff --git a/extensions/matrix/src/matrix/send/formatting.ts b/extensions/matrix/src/matrix/send/formatting.ts index 2a14ebabc07..7fea242cbed 100644 --- a/extensions/matrix/src/matrix/send/formatting.ts +++ b/extensions/matrix/src/matrix/send/formatting.ts @@ -1,5 +1,6 @@ import { getMatrixRuntime } from "../../runtime.js"; import { + markdownToMatrixHtml, resolveMatrixMentionsInMarkdown, renderMarkdownToMatrixHtmlWithMentions, type MatrixMentions, @@ -13,20 +14,45 @@ import { type MatrixRelation, type MatrixReplyRelation, type MatrixTextContent, + type MatrixTextMsgType, type MatrixThreadRelation, } from "./types.js"; const getCore = () => getMatrixRuntime(); -export function buildTextContent(body: string, relation?: MatrixRelation): MatrixTextContent { +async function renderMatrixFormattedContent(params: { + client: MatrixClient; + markdown?: string | null; + includeMentions?: boolean; +}): Promise<{ html?: string; mentions?: MatrixMentions }> { + const markdown = params.markdown ?? ""; + if (params.includeMentions === false) { + const html = markdownToMatrixHtml(markdown).trimEnd(); + return { html: html || undefined }; + } + const { html, mentions } = await renderMarkdownToMatrixHtmlWithMentions({ + markdown, + client: params.client, + }); + return { html, mentions }; +} + +export function buildTextContent( + body: string, + relation?: MatrixRelation, + opts: { + msgtype?: MatrixTextMsgType; + } = {}, +): MatrixTextContent { + const msgtype = opts.msgtype ?? MsgType.Text; return relation ? { - msgtype: MsgType.Text, + msgtype, body, "m.relates_to": relation, } : { - msgtype: MsgType.Text, + msgtype, body, }; } @@ -35,12 +61,18 @@ export async function enrichMatrixFormattedContent(params: { client: MatrixClient; content: MatrixFormattedContent; markdown?: string | null; + includeMentions?: boolean; }): Promise { - const { html, mentions } = await renderMarkdownToMatrixHtmlWithMentions({ - markdown: params.markdown ?? "", + const { html, mentions } = await renderMatrixFormattedContent({ client: params.client, + markdown: params.markdown, + includeMentions: params.includeMentions, }); - params.content["m.mentions"] = mentions; + if (mentions) { + params.content["m.mentions"] = mentions; + } else { + delete params.content["m.mentions"]; + } if (!html) { delete params.content.format; delete params.content.formatted_body; diff --git a/extensions/matrix/src/matrix/send/types.ts b/extensions/matrix/src/matrix/send/types.ts index 05919f4b1b9..9d777753874 100644 --- a/extensions/matrix/src/matrix/send/types.ts +++ b/extensions/matrix/src/matrix/send/types.ts @@ -38,6 +38,8 @@ export const EventType = { RoomMessage: "m.room.message", } as const; +export const MATRIX_OPENCLAW_FINALIZED_PREVIEW_KEY = "com.openclaw.finalized_preview" as const; + export type MatrixDirectAccountData = Record; export type MatrixReplyRelation = { @@ -110,9 +112,13 @@ export type MatrixMediaMsgType = | typeof MsgType.Video | typeof MsgType.File; +export type MatrixTextMsgType = typeof MsgType.Text | typeof MsgType.Notice; + export type MediaKind = "image" | "audio" | "video" | "document" | "unknown"; export type MatrixFormattedContent = MessageEventContent & { format?: string; formatted_body?: string; }; + +export type MatrixExtraContentFields = Record; diff --git a/extensions/matrix/src/types.ts b/extensions/matrix/src/types.ts index 725a08b5a27..ff1bb9bff9a 100644 --- a/extensions/matrix/src/types.ts +++ b/extensions/matrix/src/types.ts @@ -83,6 +83,8 @@ export type MatrixExecApprovalConfig = { target?: MatrixExecApprovalTarget; }; +export type MatrixStreamingMode = "partial" | "quiet" | "off"; + export type MatrixNetworkConfig = { /** Dangerous opt-in for trusted private/internal Matrix homeservers. */ dangerouslyAllowPrivateNetwork?: boolean; @@ -189,16 +191,20 @@ export type MatrixConfig = { /** * Streaming mode for Matrix replies. * - `"partial"`: edit a single draft message in place for the current + * assistant block as the model generates text using normal Matrix text + * messages. This preserves legacy preview-first notification behavior. + * - `"quiet"`: edit a single quiet draft notice in place for the current * assistant block as the model generates text. * - `"off"`: deliver the full reply once the model finishes. * - Use `blockStreaming: true` when you want completed assistant blocks to * stay visible as separate progress messages. When combined with - * `"partial"`, Matrix keeps a live draft for the current block and + * preview streaming, Matrix keeps a live draft for the current block and * preserves completed blocks as separate messages. - * - `true` maps to `"partial"`, `false` maps to `"off"`. + * - `true` maps to `"partial"`, `false` maps to `"off"` for backward + * compatibility. * Default: `"off"`. */ - streaming?: "partial" | "off" | boolean; + streaming?: MatrixStreamingMode | boolean; }; export type CoreConfig = { diff --git a/src/config/bundled-channel-config-metadata.generated.ts b/src/config/bundled-channel-config-metadata.generated.ts index a2eac944777..a473364860d 100644 --- a/src/config/bundled-channel-config-metadata.generated.ts +++ b/src/config/bundled-channel-config-metadata.generated.ts @@ -6692,7 +6692,7 @@ export const GENERATED_BUNDLED_CHANNEL_CONFIG_METADATA = [ anyOf: [ { type: "string", - enum: ["partial", "off"], + enum: ["partial", "quiet", "off"], }, { type: "boolean",