(fix): enforce embedding model token limit to prevent overflow (#13455)

* fix: enforce embedding model token limit to prevent 8192 overflow

- Replace EMBEDDING_APPROX_CHARS_PER_TOKEN=1 with UTF-8 byte length
  estimation (safe upper bound for tokenizer output)
- Add EMBEDDING_MODEL_MAX_TOKENS=8192 hard cap
- Add splitChunkToTokenLimit() that binary-searches for the largest
  safe split point, with surrogate pair handling
- Add enforceChunkTokenLimit() wrapper called in indexFile() after
  chunkMarkdown(), before any embedding API call
- Fixes: session files with large JSONL entries could produce chunks
  exceeding text-embedding-3-small's 8192 token limit

Tests: 2 new colocated tests in manager.embedding-token-limit.test.ts
- Verifies oversized ASCII chunks are split to <=8192 bytes each
- Verifies multibyte (emoji) content batching respects byte limits

* fix: make embedding token limit provider-aware

- Add optional maxInputTokens to EmbeddingProvider interface
- Each provider (openai, gemini, voyage) reports its own limit
- Known-limits map as fallback: openai 8192, gemini 2048, voyage 32K
- Resolution: provider field > known map > default 8192
- Backward compatible: local/llama uses fallback

* fix: enforce embedding input size limits (#13455) (thanks @rodrigouroz)

---------

Co-authored-by: Tak Hoffman <781889+Takhoffman@users.noreply.github.com>

src/memory/embedding-chunk-limits.ts

@@ -0,0 +1,30 @@
+import type { EmbeddingProvider } from "./embeddings.js";
+import { estimateUtf8Bytes, splitTextToUtf8ByteLimit } from "./embedding-input-limits.js";
+import { resolveEmbeddingMaxInputTokens } from "./embedding-model-limits.js";
+import { hashText, type MemoryChunk } from "./internal.js";
+
+export function enforceEmbeddingMaxInputTokens(
+  provider: EmbeddingProvider,
+  chunks: MemoryChunk[],
+): MemoryChunk[] {
+  const maxInputTokens = resolveEmbeddingMaxInputTokens(provider);
+  const out: MemoryChunk[] = [];
+
+  for (const chunk of chunks) {
+    if (estimateUtf8Bytes(chunk.text) <= maxInputTokens) {
+      out.push(chunk);
+      continue;
+    }
+
+    for (const text of splitTextToUtf8ByteLimit(chunk.text, maxInputTokens)) {
+      out.push({
+        startLine: chunk.startLine,
+        endLine: chunk.endLine,
+        text,
+        hash: hashText(text),
+      });
+    }
+  }
+
+  return out;
+}