addMsg now calls f.Sync() before f.Close(), matching the durability
guarantee of writeMeta and rewriteJSONL (both use WriteFileAtomic
with fsync). Without this, a power loss could leave the appended
line in the kernel page cache only — lost on reboot.
- Replace manual temp+rename in writeMeta and rewriteJSONL with the
project's standard fileutil.WriteFileAtomic. This adds fsync before
rename, which is important for flash storage on embedded devices
where power loss can leave zero-length files after an unsynced rename.
- Log a warning when readMessages skips a corrupt line, so operators
can see that data was lost after a crash instead of silently dropping it.
- Document the lossy sanitizeKey mapping (telegram:123 → telegram_123)
as an intentional tradeoff.
In SetHistory and Compact, the JSONL file was rewritten before updating
the meta file. If the process crashed between the two writes, the meta
still had a large Skip value pointing past the now-shorter JSONL file,
causing GetHistory to return empty — effectively data loss.
Reverse the order: write meta (with Skip=0) first, then rewrite JSONL.
On crash between the two writes, the old uncompacted file is still
intact and GetHistory reads from line 1, returning stale-but-complete
data. The next operation self-corrects.
A crash between the JSONL append and the meta update in addMsg can
leave meta.Count stale (e.g. file has 101 lines but meta says 100).
The previous code only reconciled when Count==0, so a nonzero stale
count was silently trusted, causing keepLast/skip to be calculated
against the wrong total.
Now TruncateHistory always counts the actual lines on disk. This is
cheap (scan without unmarshal) and TruncateHistory is not a hot path.
Address feedback from @yinwm for long-running daemon use:
- Replace sync.Map with a fixed-size sharded lock array (64 mutexes).
Keys are mapped via FNV hash, so memory is O(1) regardless of how
many sessions are created over the process lifetime.
- Increase scanner buffer cap from 1 MB to 10 MB. Tool results
(read_file on large files, web search responses) can easily exceed
1 MB. The scanner still starts at 64 KB and only grows as needed.
Address review feedback from @Zhaoyikaiii:
- Replace map[string]*sync.Mutex + separate mu with sync.Map.LoadOrStore
for simpler, lock-free session lock management.
- Add skip parameter to readMessages so callers (GetHistory, Compact)
can skip truncated lines without paying the json.Unmarshal cost.
- Add countLines helper for TruncateHistory's count reconciliation,
avoiding full deserialization when only the line count is needed.
Address file growth concern from #711 review: logical truncation via
skip offset is fast but leaves dead lines on disk indefinitely.
Compact() rewrites the JSONL file keeping only active messages, using
the same temp+rename pattern for crash safety. No-op when skip == 0.
The caller (lifecycle manager or agent loop) decides when to trigger
compaction — e.g. when skipped lines exceed active lines.
Add JSONLStore that persists sessions as .jsonl files (one message per
line) plus .meta.json for summary and truncation offset.
Key design decisions:
- Append-only writes — no full-file rewrites on AddMessage
- Logical truncation via skip offset instead of physical deletion
- Per-session mutex for safe concurrent access
- Crash recovery: malformed trailing lines are silently skipped
- Atomic metadata writes using temp+rename
Zero new dependencies — pure stdlib.
Refs #711
statxc