A crash between the JSONL append and the meta update in addMsg can
leave meta.Count stale (e.g. file has 101 lines but meta says 100).
The previous code only reconciled when Count==0, so a nonzero stale
count was silently trusted, causing keepLast/skip to be calculated
against the wrong total.
Now TruncateHistory always counts the actual lines on disk. This is
cheap (scan without unmarshal) and TruncateHistory is not a hot path.
Address feedback from @yinwm for long-running daemon use:
- Replace sync.Map with a fixed-size sharded lock array (64 mutexes).
Keys are mapped via FNV hash, so memory is O(1) regardless of how
many sessions are created over the process lifetime.
- Increase scanner buffer cap from 1 MB to 10 MB. Tool results
(read_file on large files, web search responses) can easily exceed
1 MB. The scanner still starts at 64 KB and only grows as needed.
Address review feedback from @Zhaoyikaiii:
- Replace map[string]*sync.Mutex + separate mu with sync.Map.LoadOrStore
for simpler, lock-free session lock management.
- Add skip parameter to readMessages so callers (GetHistory, Compact)
can skip truncated lines without paying the json.Unmarshal cost.
- Add countLines helper for TruncateHistory's count reconciliation,
avoiding full deserialization when only the line count is needed.
Address file growth concern from #711 review: logical truncation via
skip offset is fast but leaves dead lines on disk indefinitely.
Compact() rewrites the JSONL file keeping only active messages, using
the same temp+rename pattern for crash safety. No-op when skip == 0.
The caller (lifecycle manager or agent loop) decides when to trigger
compaction — e.g. when skipped lines exceed active lines.
Read existing sessions/*.json files, convert to JSONL format, and
rename originals to .json.migrated as backup. The migration is
idempotent — second runs skip already-migrated files.
Session keys are read from JSON content (not filenames) so that
sanitized names like telegram_123 correctly map back to telegram:123.
Cover all Store interface methods plus edge cases:
- Basic roundtrip, ordering, empty session, tool calls
- Logical truncation (keep last N, keep zero, keep more than exist)
- SetHistory replacing all + resetting skip offset
- Crash recovery with partial JSON lines
- Persistence across store instances
- Concurrent add+read (10 goroutines x 20 msgs)
- Simulated #704 race (summarizer vs main loop)
- Benchmarks for AddMessage and GetHistory (100/1000 msgs)
Add JSONLStore that persists sessions as .jsonl files (one message per
line) plus .meta.json for summary and truncation offset.
Key design decisions:
- Append-only writes — no full-file rewrites on AddMessage
- Logical truncation via skip offset instead of physical deletion
- Per-session mutex for safe concurrent access
- Crash recovery: malformed trailing lines are silently skipped
- Atomic metadata writes using temp+rename
Zero new dependencies — pure stdlib.
Refs #711
Introduce a backend-agnostic Store interface in pkg/memory/ that maps
one-to-one with the current SessionManager API. Each method is atomic
— no separate Save() call needed.
Refs #711
xiaoen