mirror of https://github.com/openclaw/openclaw.git
3b95aa8804
- Two-pass line splitting: first slice at maxChars (unchanged for Latin), then re-split only CJK-heavy segments at chunking.tokens. This preserves the original ~800-char segments for ASCII lines while keeping CJK chunks within the token budget. - Narrow surrogate-pair adjustment to CJK Extension B+ range (D840–D87E) only, so emoji surrogate pairs are not affected. Mixed CJK+emoji text is now handled consistently regardless of composition. - Add tests: emoji handling (2), Latin backward-compat long-line (1). Addresses Codex P1 (oversized CJK segments) and P2s (Latin over-splitting, emoji surrogate inconsistency). |
||
|---|---|---|
| .. | ||
| account-id.ts | ||
| boolean.ts | ||
| chunk-items.ts | ||
| cjk-chars.test.ts | ||
| cjk-chars.ts | ||
| delivery-context.test.ts | ||
| delivery-context.ts | ||
| directive-tags.test.ts | ||
| directive-tags.ts | ||
| fetch-timeout.ts | ||
| mask-api-key.test.ts | ||
| mask-api-key.ts | ||
| message-channel.test.ts | ||
| message-channel.ts | ||
| normalize-secret-input.test.ts | ||
| normalize-secret-input.ts | ||
| parse-json-compat.ts | ||
| provider-utils.ts | ||
| queue-helpers.test.ts | ||
| queue-helpers.ts | ||
| reaction-level.test.ts | ||
| reaction-level.ts | ||
| run-with-concurrency.test.ts | ||
| run-with-concurrency.ts | ||
| safe-json.ts | ||
| shell-argv.ts | ||
| transcript-tools.test.ts | ||
| transcript-tools.ts | ||
| usage-format.test.ts | ||
| usage-format.ts | ||
| utils-misc.test.ts | ||
| with-timeout.ts | ||
| zod-parse.ts | ||