Commit Graph

18 Commits

Author SHA1 Message Date
buyitsydney 4b69c6d3f1 fix(memory): add CJK/Kana/Hangul support to MMR tokenize() for diversity detection
The tokenize() function only matched [a-z0-9_]+ patterns, returning an
empty set for CJK-only text. This made Jaccard similarity always 0 (or
always 1 for two empty sets) for CJK content, effectively disabling MMR
diversity detection.

Add support for:
- CJK Unified Ideographs (U+4E00–U+9FFF, U+3400–U+4DBF)
- Hiragana (U+3040–U+309F) and Katakana (U+30A0–U+30FF)
- Hangul Syllables (U+AC00–U+D7AF) and Jamo (U+1100–U+11FF)

Characters are extracted as unigrams, and bigrams are generated only
from characters that are adjacent in the original text (no spurious
bigrams across ASCII boundaries).

Fixes #28000
2026-03-28 09:19:52 +05:30
Peter Steinberger ef1784d264 refactor: move bundled plugin policy into manifests 2026-03-27 16:40:27 +00:00
Tak Hoffman f5643544c2
Reduce lint suppressions in core tests and runtime 2026-03-27 02:11:26 -05:00
Peter Steinberger a3e73daa6b refactor: remove ollama legacy shims 2026-03-27 06:38:23 +00:00
Peter Steinberger 2f979e9be0 test: fix memory-core host type import 2026-03-27 05:38:58 +00:00
Peter Steinberger 4c27c90fc2 refactor: finish moving provider runtime into extensions 2026-03-27 05:38:58 +00:00
Peter Steinberger 64bf80d4d5 refactor: move provider runtime into extensions 2026-03-27 05:38:58 +00:00
Peter Steinberger eebce9e9c7
refactor: move memory host into sdk package 2026-03-27 04:12:04 +00:00
Peter Steinberger a9b982c954
refactor: remove memory-core engine barrel 2026-03-27 03:35:00 +00:00
Peter Steinberger f6de4cd766
refactor: remove memory-core runtime barrel 2026-03-27 02:54:23 +00:00
Peter Steinberger bd6c7969ea
refactor: extract memory host sdk package 2026-03-27 02:49:33 +00:00
Peter Steinberger 7a35bca2ec
refactor: make memory embedding adapters generic 2026-03-27 02:02:24 +00:00
Peter Steinberger dbf78de7c6
refactor: move memory engine behind plugin adapters 2026-03-27 00:47:01 +00:00
Peter Steinberger cad83db8b2
refactor: move memory engine into memory plugin 2026-03-26 23:20:35 +00:00
Peter Steinberger b96fccadb9
refactor: clean memory plugin host boundary 2026-03-26 23:02:24 +00:00
Peter Steinberger 0f5a77d058
refactor: route memory runtime through memory plugin 2026-03-26 22:30:47 +00:00
Peter Steinberger d0ce2d1044
refactor: split memory-core plugin helpers 2026-03-26 22:06:06 +00:00
Peter Steinberger 3d0050c306
refactor: add memory-core extension sources 2026-03-26 22:00:13 +00:00