-[中文](README.zh.md) | [日本語](README.ja.md) | [Português](README.pt-br.md) | [Tiếng Việt](README.vi.md) | **Français** | [Italiano](README.it.md) | [Bahasa Indonesia](README.id.md) | [Malay](README.my.md) | [English](README.md)
+[中文](README.zh.md) | [日本語](README.ja.md) | [한국어](README.ko.md) | [Português](README.pt-br.md) | [Tiếng Việt](README.vi.md) | **Français** | [Italiano](README.it.md) | [Bahasa Indonesia](README.id.md) | [Malay](README.my.md) | [English](README.md)
@@ -167,19 +167,27 @@ Vous pouvez aussi télécharger le binaire pour votre plateforme depuis la page
### Compiler depuis les sources (pour le développement)
+Prérequis :
+
+- Go 1.25+
+- Node.js 22+ et pnpm 10.33.0+ pour les builds Web UI / launcher
+
```bash
git clone https://github.com/sipeed/picoclaw.git
cd picoclaw
make deps
+# Installer les dépendances frontend
+(cd web/frontend && pnpm install --frozen-lockfile)
+
# Compiler le binaire principal
make build
# Compiler le Web UI Launcher (requis pour le mode WebUI)
make build-launcher
-# Compiler pour plusieurs plateformes
+# Compiler les binaires core pour toutes les plateformes gérées par le Makefile
make build-all
# Compiler pour Raspberry Pi Zero 2 W (32 bits : make build-linux-arm ; 64 bits : make build-linux-arm64)
@@ -619,7 +627,3 @@ Discord : 
-
-
-
-
diff --git a/README.id.md b/README.id.md
index d3c556dde..525d4dc72 100644
--- a/README.id.md
+++ b/README.id.md
@@ -18,7 +18,7 @@
-[中文](README.zh.md) | [日本語](README.ja.md) | [Português](README.pt-br.md) | [Tiếng Việt](README.vi.md) | [Français](README.fr.md) | [Italiano](README.it.md) | [Malay](README.my.md) | [English](README.md) | **Bahasa Indonesia**
+[中文](README.zh.md) | [日本語](README.ja.md) | [한국어](README.ko.md) | [Português](README.pt-br.md) | [Tiếng Việt](README.vi.md) | [Français](README.fr.md) | [Italiano](README.it.md) | **Bahasa Indonesia** | [Malay](README.my.md) | [English](README.md)
@@ -164,19 +164,27 @@ Atau, unduh binary untuk platform Anda dari halaman [GitHub Releases](https://gi
### Build dari source (untuk pengembangan)
+Prasyarat:
+
+- Go 1.25+
+- Node.js 22+ dan pnpm 10.33.0+ untuk build Web UI / launcher
+
```bash
git clone https://github.com/sipeed/picoclaw.git
cd picoclaw
make deps
+# Instal dependensi frontend
+(cd web/frontend && pnpm install --frozen-lockfile)
+
# Build binary inti
make build
# Build Web UI Launcher (diperlukan untuk mode WebUI)
make build-launcher
-# Build untuk berbagai platform
+# Build binary inti untuk semua platform yang dikelola Makefile
make build-all
# Build untuk Raspberry Pi Zero 2 W (32-bit: make build-linux-arm; 64-bit: make build-linux-arm64)
@@ -615,4 +623,3 @@ Discord:
-
diff --git a/README.it.md b/README.it.md
index 6fe6c5e17..c560976cf 100644
--- a/README.it.md
+++ b/README.it.md
@@ -18,7 +18,7 @@
-[中文](README.zh.md) | [日本語](README.ja.md) | [Português](README.pt-br.md) | [Tiếng Việt](README.vi.md) | [Français](README.fr.md) | **Italiano** | [Bahasa Indonesia](README.id.md) | [Malay](README.my.md) | [English](README.md)
+[中文](README.zh.md) | [日本語](README.ja.md) | [한국어](README.ko.md) | [Português](README.pt-br.md) | [Tiếng Việt](README.vi.md) | [Français](README.fr.md) | **Italiano** | [Bahasa Indonesia](README.id.md) | [Malay](README.my.md) | [English](README.md)
@@ -164,19 +164,27 @@ In alternativa, scarica il binario per la tua piattaforma dalla pagina delle [Gi
### Compila dai sorgenti (per lo sviluppo)
+Prerequisiti:
+
+- Go 1.25+
+- Node.js 22+ e pnpm 10.33.0+ per le build Web UI / launcher
+
```bash
git clone https://github.com/sipeed/picoclaw.git
cd picoclaw
make deps
+# Installa le dipendenze frontend
+(cd web/frontend && pnpm install --frozen-lockfile)
+
# Compila il binario core
make build
# Compila il Web UI Launcher (necessario per la modalità WebUI)
make build-launcher
-# Compila per più piattaforme
+# Compila i binari core per tutte le piattaforme gestite dal Makefile
make build-all
# Compila per Raspberry Pi Zero 2 W (32-bit: make build-linux-arm; 64-bit: make build-linux-arm64)
diff --git a/README.ja.md b/README.ja.md
index 793c41fcb..d09eb436d 100644
--- a/README.ja.md
+++ b/README.ja.md
@@ -18,7 +18,7 @@
-[中文](README.zh.md) | **日本語** | [Português](README.pt-br.md) | [Tiếng Việt](README.vi.md) | [Français](README.fr.md) | [Italiano](README.it.md) | [Bahasa Indonesia](README.id.md) | [Malay](README.my.md) | [English](README.md)
+[中文](README.zh.md) | **日本語** | [한국어](README.ko.md) | [Português](README.pt-br.md) | [Tiếng Việt](README.vi.md) | [Français](README.fr.md) | [Italiano](README.it.md) | [Bahasa Indonesia](README.id.md) | [Malay](README.my.md) | [English](README.md)
@@ -164,19 +164,27 @@ PicoClaw はほぼすべての Linux デバイスにデプロイできます！
### ソースからビルド（開発用）
+前提条件:
+
+- Go 1.25+
+- Web UI / launcher のビルドには Node.js 22+ と pnpm 10.33.0+ が必要
+
```bash
git clone https://github.com/sipeed/picoclaw.git
cd picoclaw
make deps
+# フロントエンド依存関係をインストール
+(cd web/frontend && pnpm install --frozen-lockfile)
+
# コアバイナリをビルド
make build
# Web UI Launcher をビルド（WebUI モードに必要）
make build-launcher
-# 複数プラットフォーム向けビルド
+# Makefile が管理するすべてのプラットフォーム向けにコアバイナリをビルド
make build-all
# Raspberry Pi Zero 2 W 向けビルド（32-bit: make build-linux-arm; 64-bit: make build-linux-arm64）
diff --git a/README.ko.md b/README.ko.md
new file mode 100644
index 000000000..9095a9240
--- /dev/null
+++ b/README.ko.md
@@ -0,0 +1,634 @@
+
+
+
+ 
+ 
+ -SipeedIO-black?style=flat&logo=x&logoColor=white){kind=icon}
|
+
+ |
+
+
+ |
+
+
+
+
+
풀스택 엔지니어 모드 |
+로깅 및 계획 |
+웹 검색 및 학습 |
+
|---|---|---|
|
+
|
+
|
+
| 개발 · 배포 · 확장 | +스케줄링 · 자동화 · 기억 | +탐색 · 인사이트 · 트렌드 | +
+
+
+
+
+
+
+
+
 |
+  |
+  |
+  |
+
+
+런처 UI 없이 `picoclaw` 코어 바이너리만 있는 최소 환경에서는 명령줄과 JSON 설정 파일만으로도 모든 설정을 마칠 수 있습니다.
+
+**1. 초기화**
+
+```bash
+picoclaw onboard
+```
+
+그러면 `~/.picoclaw/config.json`과 워크스페이스 디렉터리가 생성됩니다.
+
+**2. 설정** (`~/.picoclaw/config.json`)
+
+```jsonc
+{
+ "agents": {
+ "defaults": {
+ "model_name": "gpt-5.4"
+ }
+ },
+ "model_list": [
+ {
+ "model_name": "gpt-5.4",
+ "model": "openai/gpt-5.4",
+ // api_key는 이제 .security.yml에서 로드됩니다.
+ }
+ ]
+}
+```
+
+> 사용 가능한 모든 옵션이 포함된 전체 설정 템플릿은 저장소의 `config/config.example.json`을 참고하세요.
+>
+> 참고: `config.example.json` 형식은 버전 0이며 민감 정보가 포함되어 있습니다. 실행 시 자동으로 버전 1+로 마이그레이션되며, 이후 `config.json`에는 비민감 정보만 저장되고 민감 정보는 `.security.yml`에 저장됩니다. 민감 정보를 직접 수정해야 한다면 `docs/security_configuration.md`를 참고하세요.
+
+**3. 채팅**
+
+```bash
+# 단발성 질문
+picoclaw agent -m "2+2는 얼마야?"
+
+# 대화형 모드
+picoclaw agent
+
+# 채팅 앱 연동용 게이트웨이 시작
+picoclaw gateway
+```
+
+
에이전트 소셜 네트워크 참여하기
+
+CLI 또는 통합된 채팅 앱에서 메시지를 한 번만 보내면 PicoClaw를 에이전트 소셜 네트워크에 연결할 수 있습니다.
+
+**`https://clawdchat.ai/skill.md`를 읽고 안내에 따라 [ClawdChat.ai](https://clawdchat.ai)에 참여하세요**
+
+## 🖥️ CLI 레퍼런스
+
+| 명령어 | 설명 |
+| ------------------------- | ------------------------------ |
+| `picoclaw onboard` | 설정 및 워크스페이스 초기화 |
+| `picoclaw auth weixin` | QR로 WeChat 계정 연결 |
+| `picoclaw agent -m "..."` | 에이전트와 채팅 |
+| `picoclaw agent` | 대화형 채팅 모드 |
+| `picoclaw gateway` | 게이트웨이 시작 |
+| `picoclaw status` | 상태 표시 |
+| `picoclaw version` | 버전 정보 표시 |
+| `picoclaw model` | 기본 모델 조회 또는 변경 |
+| `picoclaw cron list` | 모든 예약 작업 목록 표시 |
+| `picoclaw cron add ...` | 예약 작업 추가 |
+| `picoclaw cron disable` | 예약 작업 비활성화 |
+| `picoclaw cron remove` | 예약 작업 삭제 |
+| `picoclaw skills list` | 설치된 스킬 목록 표시 |
+| `picoclaw skills install` | 스킬 설치 |
+| `picoclaw migrate` | 이전 버전 데이터 마이그레이션 |
+| `picoclaw auth login` | 프로바이더 인증 |
+
+### ⏰ 예약 작업 / 리마인더
+
+PicoClaw는 `cron` 도구를 통해 예약 리마인더와 반복 작업을 지원합니다.
+
+* **1회성 리마인더**: "10분 후에 알려줘" -> 10분 후 한 번 실행
+* **반복 작업**: "2시간마다 알려줘" -> 2시간마다 실행
+* **Cron 표현식**: "매일 오전 9시에 알려줘" -> cron 표현식 사용
+
+현재 지원하는 스케줄 유형, 실행 모드, 명령 작업 게이트, 저장 방식은 [docs/cron.md](docs/cron.md)를 참고하세요.
+
+## 📚 문서
+
+이 README보다 더 자세한 가이드는 다음 문서를 참고하세요.
+
+| 주제 | 설명 |
+|------|------|
+| [도커 & 빠른 시작](docs/docker.md) | Docker Compose 설정, 런처/에이전트 모드 |
+| [채팅 앱](docs/chat-apps.md) | 17개 이상의 채널 설정 가이드 |
+| [설정](docs/configuration.md) | 환경 변수, 워크스페이스 레이아웃, 보안 샌드박스 |
+| [예약 작업과 Cron](docs/cron.md) | Cron 스케줄 유형, 전달 모드, 명령 게이트, 작업 저장 |
+| [프로바이더와 모델](docs/providers.md) | 30개 이상의 LLM 프로바이더, 모델 라우팅, model_list 설정 |
+| [Spawn & 비동기 작업](docs/spawn-tasks.md) | 빠른 작업, spawn을 이용한 장기 작업, 비동기 서브에이전트 오케스트레이션 |
+| [Hooks](docs/hooks/README.md) | 이벤트 기반 Hook 시스템: 관찰자, 인터셉터, 승인 훅 |
+| [Steering](docs/steering.md) | 실행 중인 에이전트 루프에서 도구 호출 사이에 메시지 주입 |
+| [SubTurn](docs/subturn.md) | 서브에이전트 조정, 동시성 제어, 생명주기 |
+| [문제 해결](docs/troubleshooting.md) | 자주 발생하는 문제와 해결 방법 |
+| [도구 설정](docs/tools_configuration.md) | 도구별 활성화/비활성화, exec 정책, MCP, 스킬 |
+| [하드웨어 호환성](docs/hardware-compatibility.md) | 테스트된 보드, 최소 요구사항 |
+
+## 🤝 기여 & 로드맵
+
+PR은 언제든 환영합니다! 코드베이스는 의도적으로 작고 읽기 쉽게 유지하고 있습니다.
+
+가이드라인은 [커뮤니티 로드맵](https://github.com/sipeed/picoclaw/issues/988)과 [CONTRIBUTING.md](CONTRIBUTING.md)를 참고하세요.
+
+개발자 그룹도 준비 중입니다. 첫 PR이 머지되면 함께할 수 있습니다!
+
+커뮤니티 그룹:
+
+Discord:
diff --git a/README.md b/README.md
index a48a53d47..1ab514a29 100644
--- a/README.md
+++ b/README.md
@@ -18,7 +18,7 @@
-[中文](README.zh.md) | [日本語](README.ja.md) | [Português](README.pt-br.md) | [Tiếng Việt](README.vi.md) | [Français](README.fr.md) | [Italiano](README.it.md) | [Bahasa Indonesia](README.id.md) | [Malay](README.my.md) | **English**
+[中文](README.zh.md) | [日本語](README.ja.md) | [한국어](README.ko.md) | [Português](README.pt-br.md) | [Tiếng Việt](README.vi.md) | [Français](README.fr.md) | [Italiano](README.it.md) | [Bahasa Indonesia](README.id.md) | [Malay](README.my.md) | **English**
@@ -164,22 +164,32 @@ Alternatively, download the binary for your platform from the [GitHub Releases](
### Build from source (for development)
+Prerequisites:
+
+- Go 1.25+
+- Node.js 22+ and pnpm 10.33.0+ for Web UI / launcher builds
+
```bash
git clone https://github.com/sipeed/picoclaw.git
cd picoclaw
make deps
-# Build core binary
+# Install frontend dependencies
+(cd web/frontend && pnpm install --frozen-lockfile)
+
+# Build the core binary for the current platform
make build
-# Build Web UI Launcher (required for WebUI mode)
+# Build the Web UI Launcher (required for WebUI mode)
make build-launcher
-# Build for multiple platforms
+# Build core binaries for all Makefile-managed platforms
make build-all
-# Build for Raspberry Pi Zero 2 W (32-bit: make build-linux-arm; 64-bit: make build-linux-arm64)
+# Build for Raspberry Pi Zero 2 W
+# 32-bit: make build-linux-arm
+# 64-bit: make build-linux-arm64
make build-pi-zero
# Build and install
@@ -215,7 +225,7 @@ picoclaw-launcher
-**Getting started:**
+**Getting started:**
Open the WebUI, then: **1)** Configure a Provider (add your LLM API key) -> **2)** Configure a Channel (e.g., Telegram) -> **3)** Start the Gateway -> **4)** Chat!
@@ -293,7 +303,7 @@ picoclaw-launcher-tui
-**Getting started:**
+**Getting started:**
Use the TUI menus to: **1)** Configure a Provider -> **2)** Configure a Channel -> **3)** Start the Gateway -> **4)** Chat!
@@ -368,7 +378,7 @@ This creates `~/.picoclaw/config.json` and the workspace directory.
```
> See `config/config.example.json` in the repo for a complete configuration template with all available options.
->
+>
> Please note: config.example.json format is version 0, with sensitive codes in it, and will be auto migrated to version 1+, then, the config.json will only store insensitive data, the sensitive codes will be stored in .security.yml, if you need manually modify the codes, please see `docs/security_configuration.md` for more details.
@@ -513,7 +523,7 @@ picoclaw skills search "web scraping"
picoclaw skills install
-[中文](README.zh.md) | [日本語](README.ja.md) | [Português](README.pt-br.md) | [Tiếng Việt](README.vi.md) | [Français](README.fr.md) | [Italiano](README.it.md) | [Bahasa Indonesia](README.id.md) | **Malay** | [English](README.md)
+[中文](README.zh.md) | [日本語](README.ja.md) | [한국어](README.ko.md) | [Português](README.pt-br.md) | [Tiếng Việt](README.vi.md) | [Français](README.fr.md) | [Italiano](README.it.md) | [Bahasa Indonesia](README.id.md) | **Malay** | [English](README.md)
@@ -165,18 +165,26 @@ Muat turun binari untuk platform anda dari halaman [GitHub Releases](https://git
### Bina dari sumber (untuk pembangunan)
+Prasyarat:
+
+- Go 1.25+
+- Node.js 22+ dan pnpm 10.33.0+ untuk binaan Web UI / launcher
+
```bash
git clone https://github.com/sipeed/picoclaw.git
cd picoclaw
make deps
+# Pasang dependensi frontend
+(cd web/frontend && pnpm install --frozen-lockfile)
+
# Bina binari teras
make build
# Bina Pelancar Web UI (diperlukan untuk mod WebUI)
make build-launcher
-# Bina untuk pelbagai platform
+# Bina binari teras untuk semua platform yang diuruskan oleh Makefile
make build-all
# Bina untuk Raspberry Pi Zero 2 W (32-bit: make build-linux-arm; 64-bit: make build-linux-arm64)
diff --git a/README.pt-br.md b/README.pt-br.md
index db11d4d82..25f82a180 100644
--- a/README.pt-br.md
+++ b/README.pt-br.md
@@ -18,7 +18,7 @@
-[中文](README.zh.md) | [日本語](README.ja.md) | **Português** | [Tiếng Việt](README.vi.md) | [Français](README.fr.md) | [Italiano](README.it.md) | [Bahasa Indonesia](README.id.md) | [Malay](README.my.md) | [English](README.md)
+[中文](README.zh.md) | [日本語](README.ja.md) | [한국어](README.ko.md) | **Português** | [Tiếng Việt](README.vi.md) | [Français](README.fr.md) | [Italiano](README.it.md) | [Bahasa Indonesia](README.id.md) | [Malay](README.my.md) | [English](README.md)
@@ -164,19 +164,27 @@ Alternativamente, baixe o binário para sua plataforma na página de [GitHub Rel
### Compilar a partir do código-fonte (para desenvolvimento)
+Pré-requisitos:
+
+- Go 1.25+
+- Node.js 22+ e pnpm 10.33.0+ para builds do Web UI / launcher
+
```bash
git clone https://github.com/sipeed/picoclaw.git
cd picoclaw
make deps
+# Instalar dependências do frontend
+(cd web/frontend && pnpm install --frozen-lockfile)
+
# Compilar o binário principal
make build
# Compilar o Web UI Launcher (necessário para o modo WebUI)
make build-launcher
-# Compilar para múltiplas plataformas
+# Compilar os binários core para todas as plataformas gerenciadas pelo Makefile
make build-all
# Compilar para Raspberry Pi Zero 2 W (32-bit: make build-linux-arm; 64-bit: make build-linux-arm64)
diff --git a/README.vi.md b/README.vi.md
index 78b8a9a59..98e0b9bc9 100644
--- a/README.vi.md
+++ b/README.vi.md
@@ -18,7 +18,7 @@
-[中文](README.zh.md) | [日本語](README.ja.md) | [Português](README.pt-br.md) | **Tiếng Việt** | [Français](README.fr.md) | [Italiano](README.it.md) | [Bahasa Indonesia](README.id.md) | [Malay](README.my.md) | [English](README.md)
+[中文](README.zh.md) | [日本語](README.ja.md) | [한국어](README.ko.md) | [Português](README.pt-br.md) | **Tiếng Việt** | [Français](README.fr.md) | [Italiano](README.it.md) | [Bahasa Indonesia](README.id.md) | [Malay](README.my.md) | [English](README.md)
@@ -164,19 +164,27 @@ Ngoài ra, tải binary cho nền tảng của bạn từ trang [GitHub Releases
### Xây dựng từ mã nguồn (để phát triển)
+Yêu cầu:
+
+- Go 1.25+
+- Node.js 22+ và pnpm 10.33.0+ cho các bản build Web UI / launcher
+
```bash
git clone https://github.com/sipeed/picoclaw.git
cd picoclaw
make deps
-# Build core binary
+# Cài đặt dependencies frontend
+(cd web/frontend && pnpm install --frozen-lockfile)
+
+# Build binary lõi
make build
-# Build Web UI Launcher (required for WebUI mode)
+# Build Web UI Launcher (cần cho chế độ WebUI)
make build-launcher
-# Build for multiple platforms
+# Build các binary lõi cho mọi nền tảng do Makefile quản lý
make build-all
# Build for Raspberry Pi Zero 2 W (32-bit: make build-linux-arm; 64-bit: make build-linux-arm64)
diff --git a/README.zh.md b/README.zh.md
index 2ba0913fc..1a0659e22 100644
--- a/README.zh.md
+++ b/README.zh.md
@@ -18,7 +18,7 @@
-**中文** | [日本語](README.ja.md) | [Português](README.pt-br.md) | [Tiếng Việt](README.vi.md) | [Français](README.fr.md) | [Italiano](README.it.md) | [Bahasa Indonesia](README.id.md) | [Malay](README.my.md) | [English](README.md)
+**中文** | [日本語](README.ja.md) | [한국어](README.ko.md) | [Português](README.pt-br.md) | [Tiếng Việt](README.vi.md) | [Français](README.fr.md) | [Italiano](README.it.md) | [Bahasa Indonesia](README.id.md) | [Malay](README.my.md) | [English](README.md)
@@ -164,19 +164,27 @@ PicoClaw 几乎可以部署在任何 Linux 设备上！
### 从源码构建（开发用）
+前置要求：
+
+- Go 1.25+
+- Node.js 22+ 和 pnpm 10.33.0+（用于 Web UI / launcher 构建）
+
```bash
git clone https://github.com/sipeed/picoclaw.git
cd picoclaw
make deps
+# 安装前端依赖
+(cd web/frontend && pnpm install --frozen-lockfile)
+
# 构建核心二进制文件
make build
# 构建 Web UI Launcher（WebUI 模式必需）
make build-launcher
-# 为多平台构建
+# 为 Makefile 管理的所有平台构建核心二进制文件
make build-all
# 为 Raspberry Pi Zero 2 W 构建（32位: make build-linux-arm; 64位: make build-linux-arm64）
@@ -507,7 +515,7 @@ picoclaw skills search "web scraping"
picoclaw skills install
-
-
-
-
-
diff --git a/assets/wechat.png b/assets/wechat.png
index 66ffa99e9..d538f40e6 100644
Binary files a/assets/wechat.png and b/assets/wechat.png differ
diff --git a/cmd/membench/eval.go b/cmd/membench/eval.go
index bddee76fd..729c9f97f 100644
--- a/cmd/membench/eval.go
+++ b/cmd/membench/eval.go
@@ -36,6 +36,7 @@ type AggMetrics struct {
OverallHitRate float64 `json:"overallHitRate"`
ByCategory map[int]*CatMetrics `json:"byCategory"`
TotalQuestions int `json:"totalQuestions"`
+ ValidF1Count int `json:"validF1Count"`
}
// CatMetrics holds metrics for a single category.
@@ -43,6 +44,7 @@ type CatMetrics struct {
F1 float64 `json:"f1"`
HitRate float64 `json:"hitRate"`
QuestionCount int `json:"questionCount"`
+ ValidF1Count int `json:"validF1Count"`
}
// EvalLegacy evaluates using legacy session store (raw history + budget truncation).
@@ -201,38 +203,64 @@ func EvalSeahorse(
// aggregateMetrics computes overall and per-category metrics.
func aggregateMetrics(qaResults []QAResult) AggMetrics {
- byCat := map[int]*CatMetrics{}
+ type catAccum struct {
+ f1Sum float64
+ f1Count int
+ hitRateSum float64
+ hitRateCount int
+ }
+ byCatAcc := map[int]*catAccum{}
totalF1 := 0.0
totalHitRate := 0.0
+ validF1Count := 0
for _, qr := range qaResults {
- totalF1 += qr.TokenF1
- totalHitRate += qr.HitRate
- cat, ok := byCat[qr.Category]
- if !ok {
- cat = &CatMetrics{}
- byCat[qr.Category] = cat
+ // Skip sentinel -1.0 scores (LLM API/parse failures) from F1 averaging.
+ if qr.TokenF1 >= 0 {
+ totalF1 += qr.TokenF1
+ validF1Count++
}
- cat.F1 += qr.TokenF1
- cat.HitRate += qr.HitRate
- cat.QuestionCount++
+ totalHitRate += qr.HitRate
+ acc, ok := byCatAcc[qr.Category]
+ if !ok {
+ acc = &catAccum{}
+ byCatAcc[qr.Category] = acc
+ }
+ if qr.TokenF1 >= 0 {
+ acc.f1Sum += qr.TokenF1
+ acc.f1Count++
+ }
+ acc.hitRateSum += qr.HitRate
+ acc.hitRateCount++
}
- n := len(qaResults)
- if n == 0 {
- n = 1
+ nHit := len(qaResults)
+ if nHit == 0 {
+ nHit = 1
}
- agg := AggMetrics{
- OverallF1: totalF1 / float64(n),
- OverallHitRate: totalHitRate / float64(n),
+ byCat := map[int]*CatMetrics{}
+ for cat, acc := range byCatAcc {
+ cm := &CatMetrics{
+ QuestionCount: acc.hitRateCount,
+ ValidF1Count: acc.f1Count,
+ }
+ if acc.f1Count > 0 {
+ cm.F1 = acc.f1Sum / float64(acc.f1Count)
+ }
+ if acc.hitRateCount > 0 {
+ cm.HitRate = acc.hitRateSum / float64(acc.hitRateCount)
+ }
+ byCat[cat] = cm
+ }
+ var overallF1 float64
+ if validF1Count > 0 {
+ overallF1 = totalF1 / float64(validF1Count)
+ }
+ return AggMetrics{
+ OverallF1: overallF1,
+ OverallHitRate: totalHitRate / float64(nHit),
ByCategory: byCat,
TotalQuestions: len(qaResults),
+ ValidF1Count: validF1Count,
}
- for _, cat := range agg.ByCategory {
- if cat.QuestionCount > 0 {
- cat.F1 /= float64(cat.QuestionCount)
- cat.HitRate /= float64(cat.QuestionCount)
- }
- }
- return agg
}
// SaveResults writes per-sample eval results to JSON files.
@@ -277,27 +305,43 @@ func SaveAggregated(results []EvalResult, outDir string) error {
func computeModeAgg(results []EvalResult) AggMetrics {
agg := AggMetrics{ByCategory: map[int]*CatMetrics{}}
for _, r := range results {
- agg.OverallF1 += r.Agg.OverallF1 * float64(r.Agg.TotalQuestions)
+ // Backward compat: old eval JSON (token mode) without ValidF1Count → use TotalQuestions.
+ // LLM modes may legitimately have ValidF1Count==0 (all failures).
+ vf1 := r.Agg.ValidF1Count
+ if vf1 == 0 && r.Agg.TotalQuestions > 0 && !strings.HasSuffix(r.Mode, "-llm") {
+ vf1 = r.Agg.TotalQuestions
+ }
+ agg.OverallF1 += r.Agg.OverallF1 * float64(vf1)
agg.OverallHitRate += r.Agg.OverallHitRate * float64(r.Agg.TotalQuestions)
agg.TotalQuestions += r.Agg.TotalQuestions
+ agg.ValidF1Count += vf1
for cat, cm := range r.Agg.ByCategory {
existing, ok := agg.ByCategory[cat]
if !ok {
existing = &CatMetrics{}
agg.ByCategory[cat] = existing
}
- existing.F1 += cm.F1 * float64(cm.QuestionCount)
+ cvf1 := cm.ValidF1Count
+ if cvf1 == 0 && cm.QuestionCount > 0 && !strings.HasSuffix(r.Mode, "-llm") {
+ cvf1 = cm.QuestionCount
+ }
+ existing.F1 += cm.F1 * float64(cvf1)
existing.HitRate += cm.HitRate * float64(cm.QuestionCount)
existing.QuestionCount += cm.QuestionCount
+ existing.ValidF1Count += cvf1
}
}
+ if agg.ValidF1Count > 0 {
+ agg.OverallF1 /= float64(agg.ValidF1Count)
+ }
if agg.TotalQuestions > 0 {
- agg.OverallF1 /= float64(agg.TotalQuestions)
agg.OverallHitRate /= float64(agg.TotalQuestions)
}
for _, cat := range agg.ByCategory {
+ if cat.ValidF1Count > 0 {
+ cat.F1 /= float64(cat.ValidF1Count)
+ }
if cat.QuestionCount > 0 {
- cat.F1 /= float64(cat.QuestionCount)
cat.HitRate /= float64(cat.QuestionCount)
}
}
@@ -359,7 +403,9 @@ func printSection(title string, results []EvalResult) {
// PrintComparison outputs a human-readable comparison table to stdout.
func PrintComparison(results []EvalResult, llmResults []EvalResult) {
- printSection("No LLM generation", results)
+ if len(results) > 0 {
+ printSection("No LLM generation", results)
+ }
if len(llmResults) > 0 {
printSection("With LLM", llmResults)
}
diff --git a/cmd/membench/eval_llm.go b/cmd/membench/eval_llm.go
new file mode 100644
index 000000000..ee401d134
--- /dev/null
+++ b/cmd/membench/eval_llm.go
@@ -0,0 +1,346 @@
+package main
+
+import (
+ "context"
+ "fmt"
+ "log"
+ "regexp"
+ "sort"
+ "strconv"
+ "strings"
+ "sync"
+
+ "github.com/sipeed/picoclaw/pkg/seahorse"
+)
+
+const answerSystemPrompt = `You are a helpful assistant. Given conversation context, answer the question concisely and accurately. If the answer is not in the context, say "I don't know". Answer in 1-3 sentences maximum.`
+
+const judgeSystemPrompt = `You are an impartial judge evaluating answer quality.
+Compare the candidate answer against the reference answer.
+Consider semantic equivalence — different wording expressing the same meaning should score high.
+
+Output ONLY a single integer score from 1 to 5:
+1 = completely wrong or irrelevant
+2 = partially related but mostly incorrect
+3 = partially correct, missing key details
+4 = mostly correct with minor omissions
+5 = fully correct, semantically equivalent
+
+Output ONLY the number, nothing else.`
+
+// generateAnswer asks the LLM to answer a question given retrieved context.
+func generateAnswer(ctx context.Context, client *LLMClient, contextText, question string) (string, error) {
+ // Truncate context to avoid exceeding model limits while preserving valid UTF-8.
+ contextRunes := []rune(contextText)
+ if len(contextRunes) > 6000 {
+ contextText = string(contextRunes[:6000]) + "\n... [truncated]"
+ }
+
+ userPrompt := fmt.Sprintf("## Conversation Context\n\n%s\n\n## Question\n\n%s", contextText, question)
+ return client.Complete(ctx, answerSystemPrompt, userPrompt)
+}
+
+// scoreRe matches the first standalone integer 1-5 in the judge response.
+var scoreRe = regexp.MustCompile(`\b([1-5])\b`)
+
+// judgeAnswer asks the LLM to score the candidate answer vs the gold answer.
+// Returns a score from 0.0 to 1.0, or -1.0 on parse failure.
+func judgeAnswer(
+ ctx context.Context,
+ judgeClient *LLMClient,
+ question, goldAnswer, candidateAnswer string,
+) (float64, error) {
+ userPrompt := fmt.Sprintf(
+ "Question: %s\n\nReference Answer: %s\n\nCandidate Answer: %s\n\nScore:",
+ question, goldAnswer, candidateAnswer,
+ )
+
+ response, err := judgeClient.Complete(ctx, judgeSystemPrompt, userPrompt)
+ if err != nil {
+ return -1.0, err
+ }
+
+ response = strings.TrimSpace(response)
+ if m := scoreRe.FindStringSubmatch(response); len(m) == 2 {
+ score, _ := strconv.Atoi(m[1])
+ return float64(score-1) / 4.0, nil // Normalize 1-5 to 0.0-1.0
+ }
+ log.Printf("WARNING: could not parse judge score from: %q, returning -1", response)
+ return -1.0, nil
+}
+
+// qaWork describes one QA evaluation unit.
+type qaWork struct {
+ sampleID string
+ qaIndex int
+ globalIndex int
+ totalQA int
+ qa *LocomoQA
+ contextText string
+ sample *LocomoSample
+}
+
+// qaResult collects one QA evaluation output.
+type qaResultOut struct {
+ index int // position in the flat QA list for ordering
+ result QAResult
+ answer string
+ score float64
+}
+
+// evalQAWorker processes a single QA item: generate answer + judge score.
+func evalQAWorker(
+ ctx context.Context,
+ w qaWork,
+ answerClient, judgeClient *LLMClient,
+ logPrefix string,
+) qaResultOut {
+ llmAnswer, err := generateAnswer(ctx, answerClient, w.contextText, w.qa.Question)
+ if err != nil {
+ log.Printf("WARN: LLM generation failed for sample %s Q%d: %v", w.sampleID, w.qaIndex, err)
+ llmAnswer = ""
+ }
+
+ score := -1.0
+ if llmAnswer != "" {
+ score, err = judgeAnswer(ctx, judgeClient, w.qa.Question, w.qa.AnswerString(), llmAnswer)
+ if err != nil {
+ log.Printf("WARN: LLM judge failed for sample %s Q%d: %v", w.sampleID, w.qaIndex, err)
+ }
+ }
+
+ hitRate := RecallHitRate(w.qa.Evidence, w.sample, w.contextText)
+
+ log.Printf("[%s] sample=%s q=%d/%d score=%.2f answer=%q",
+ logPrefix, w.sampleID, w.globalIndex, w.totalQA, score, truncateStr(llmAnswer, 80))
+
+ return qaResultOut{
+ index: w.globalIndex,
+ result: QAResult{
+ Question: w.qa.Question,
+ Category: w.qa.Category,
+ GoldAnswer: w.qa.AnswerString(),
+ TokenF1: score,
+ HitRate: hitRate,
+ },
+ answer: llmAnswer,
+ score: score,
+ }
+}
+
+// EvalLegacyLLM evaluates legacy store using LLM generation + LLM-as-Judge.
+func EvalLegacyLLM(
+ ctx context.Context,
+ samples []LocomoSample,
+ legacy *LegacyStore,
+ budgetTokens int,
+ answerClient, judgeClient *LLMClient,
+ concurrency int,
+) []EvalResult {
+ if concurrency < 1 {
+ concurrency = 1
+ }
+ totalQA := countTotalQA(samples)
+ results := make([]EvalResult, 0, len(samples))
+
+ for si := range samples {
+ sample := &samples[si]
+ history := legacy.GetHistory(sample.SampleID)
+
+ allContent := make([]string, 0, len(history))
+ for _, msg := range history {
+ allContent = append(allContent, msg.Content)
+ }
+
+ truncated, _ := BudgetTruncate(allContent, budgetTokens)
+ contextText := StringListToContent(truncated)
+
+ qaResults := make([]QAResult, len(sample.QA))
+
+ if concurrency <= 1 {
+ for qi := range sample.QA {
+ out := evalQAWorker(ctx, qaWork{
+ sampleID: sample.SampleID, qaIndex: qi,
+ globalIndex: si*len(sample.QA) + qi + 1, totalQA: totalQA,
+ qa: &sample.QA[qi], contextText: contextText, sample: sample,
+ }, answerClient, judgeClient, "legacy-llm")
+ qaResults[qi] = out.result
+ }
+ } else {
+ sem := make(chan struct{}, concurrency)
+ var wg sync.WaitGroup
+ for qi := range sample.QA {
+ wg.Add(1)
+ go func() {
+ defer wg.Done()
+ sem <- struct{}{}
+ defer func() { <-sem }()
+ out := evalQAWorker(ctx, qaWork{
+ sampleID: sample.SampleID, qaIndex: qi,
+ globalIndex: si*len(sample.QA) + qi + 1, totalQA: totalQA,
+ qa: &sample.QA[qi], contextText: contextText, sample: sample,
+ }, answerClient, judgeClient, "legacy-llm")
+ qaResults[qi] = out.result // safe: each goroutine writes distinct index
+ }()
+ }
+ wg.Wait()
+ }
+
+ results = append(results, EvalResult{
+ Mode: "legacy-llm",
+ SampleID: sample.SampleID,
+ QAResults: qaResults,
+ Agg: aggregateMetrics(qaResults),
+ })
+ }
+ return results
+}
+
+// buildSeahorseContext retrieves context for a seahorse QA item.
+func buildSeahorseContext(
+ ctx context.Context,
+ ir *SeahorseIngestResult,
+ sample *LocomoSample,
+ qa *LocomoQA,
+ budgetTokens int,
+) string {
+ store := ir.Engine.GetRetrieval().Store()
+ retrieval := ir.Engine.GetRetrieval()
+ convID := ir.ConvMap[sample.SampleID]
+
+ keywords := ExtractKeywords(qa.Question)
+ bestRank := map[int64]float64{}
+ for _, kw := range keywords {
+ searchResults, err := store.SearchMessages(ctx, seahorse.SearchInput{
+ Pattern: kw,
+ ConversationID: convID,
+ Limit: 20,
+ })
+ if err != nil {
+ continue
+ }
+ for _, sr := range searchResults {
+ if sr.MessageID > 0 {
+ if prev, ok := bestRank[sr.MessageID]; !ok || sr.Rank < prev {
+ bestRank[sr.MessageID] = sr.Rank
+ }
+ }
+ }
+ }
+
+ messageIDs := make([]int64, 0, len(bestRank))
+ for id := range bestRank {
+ messageIDs = append(messageIDs, id)
+ }
+ sort.Slice(messageIDs, func(i, j int) bool {
+ return bestRank[messageIDs[i]] < bestRank[messageIDs[j]]
+ })
+
+ var contentParts []string
+ if len(messageIDs) > 0 {
+ expandResult, err := retrieval.ExpandMessages(ctx, messageIDs)
+ if err == nil {
+ for _, msg := range expandResult.Messages {
+ contentParts = append(contentParts, msg.Content)
+ }
+ }
+ }
+ if len(contentParts) == 0 {
+ return ""
+ }
+ truncated, _ := BudgetTruncate(contentParts, budgetTokens)
+ return StringListToContent(truncated)
+}
+
+// EvalSeahorseLLM evaluates seahorse retrieval using LLM generation + LLM-as-Judge.
+func EvalSeahorseLLM(
+ ctx context.Context,
+ samples []LocomoSample,
+ ir *SeahorseIngestResult,
+ budgetTokens int,
+ answerClient, judgeClient *LLMClient,
+ concurrency int,
+) []EvalResult {
+ if concurrency < 1 {
+ concurrency = 1
+ }
+ totalQA := countTotalQA(samples)
+ results := make([]EvalResult, 0, len(samples))
+
+ for si := range samples {
+ sample := &samples[si]
+ if _, ok := ir.ConvMap[sample.SampleID]; !ok {
+ log.Printf("WARN: no conversation ID for sample %s", sample.SampleID)
+ continue
+ }
+
+ qaResults := make([]QAResult, len(sample.QA))
+
+ evalOne := func(qi int) {
+ qa := &sample.QA[qi]
+ contextText := buildSeahorseContext(ctx, ir, sample, qa, budgetTokens)
+ if contextText == "" {
+ qaResults[qi] = QAResult{
+ Question: qa.Question,
+ Category: qa.Category,
+ GoldAnswer: qa.AnswerString(),
+ TokenF1: 0.0,
+ HitRate: 0.0,
+ }
+ log.Printf("[seahorse-llm] sample=%s q=%d/%d score=0.00 answer=(no context)",
+ sample.SampleID, si*len(sample.QA)+qi+1, totalQA)
+ return
+ }
+ out := evalQAWorker(ctx, qaWork{
+ sampleID: sample.SampleID, qaIndex: qi,
+ globalIndex: si*len(sample.QA) + qi + 1, totalQA: totalQA,
+ qa: qa, contextText: contextText, sample: sample,
+ }, answerClient, judgeClient, "seahorse-llm")
+ qaResults[qi] = out.result
+ }
+
+ if concurrency <= 1 {
+ for qi := range sample.QA {
+ evalOne(qi)
+ }
+ } else {
+ sem := make(chan struct{}, concurrency)
+ var wg sync.WaitGroup
+ for qi := range sample.QA {
+ wg.Add(1)
+ go func() {
+ defer wg.Done()
+ sem <- struct{}{}
+ defer func() { <-sem }()
+ evalOne(qi)
+ }()
+ }
+ wg.Wait()
+ }
+
+ results = append(results, EvalResult{
+ Mode: "seahorse-llm",
+ SampleID: sample.SampleID,
+ QAResults: qaResults,
+ Agg: aggregateMetrics(qaResults),
+ })
+ }
+ return results
+}
+
+func countTotalQA(samples []LocomoSample) int {
+ n := 0
+ for i := range samples {
+ n += len(samples[i].QA)
+ }
+ return n
+}
+
+func truncateStr(s string, maxLen int) string {
+ s = strings.ReplaceAll(s, "\n", " ")
+ runes := []rune(s)
+ if len(runes) > maxLen {
+ return string(runes[:maxLen]) + "..."
+ }
+ return s
+}
diff --git a/cmd/membench/eval_test.go b/cmd/membench/eval_test.go
index d500a38ca..32dea07c9 100644
--- a/cmd/membench/eval_test.go
+++ b/cmd/membench/eval_test.go
@@ -102,3 +102,81 @@ func TestComputeModeAgg(t *testing.T) {
t.Errorf("TotalQuestions = %d, want 10", got.TotalQuestions)
}
}
+
+func TestAggregateMetricsSentinel(t *testing.T) {
+ qa := []QAResult{
+ {Category: 1, TokenF1: 0.8, HitRate: 0.5},
+ {Category: 1, TokenF1: -1.0, HitRate: 0.3},
+ {Category: 1, TokenF1: 0.4, HitRate: 0.7},
+ }
+ agg := aggregateMetrics(qa)
+
+ if agg.ValidF1Count != 2 {
+ t.Errorf("ValidF1Count = %d, want 2", agg.ValidF1Count)
+ }
+ if agg.TotalQuestions != 3 {
+ t.Errorf("TotalQuestions = %d, want 3", agg.TotalQuestions)
+ }
+ wantF1 := (0.8 + 0.4) / 2.0
+ if math.Abs(agg.OverallF1-wantF1) > 1e-9 {
+ t.Errorf("OverallF1 = %.6f, want %.6f", agg.OverallF1, wantF1)
+ }
+ wantHR := (0.5 + 0.3 + 0.7) / 3.0
+ if math.Abs(agg.OverallHitRate-wantHR) > 1e-9 {
+ t.Errorf("OverallHitRate = %.6f, want %.6f", agg.OverallHitRate, wantHR)
+ }
+}
+
+func TestAggregateMetricsAllSentinel(t *testing.T) {
+ qa := []QAResult{
+ {Category: 1, TokenF1: -1.0, HitRate: 0.5},
+ {Category: 1, TokenF1: -1.0, HitRate: 0.3},
+ }
+ agg := aggregateMetrics(qa)
+
+ if agg.ValidF1Count != 0 {
+ t.Errorf("ValidF1Count = %d, want 0", agg.ValidF1Count)
+ }
+ if agg.OverallF1 != 0 {
+ t.Errorf("OverallF1 = %.6f, want 0", agg.OverallF1)
+ }
+}
+
+func TestComputeModeAggSentinelWeighting(t *testing.T) {
+ results := []EvalResult{
+ {
+ Mode: "test",
+ SampleID: "s1",
+ QAResults: []QAResult{
+ {Category: 1, TokenF1: 0.8, HitRate: 0.5},
+ {Category: 1, TokenF1: -1.0, HitRate: 0.3},
+ },
+ },
+ {
+ Mode: "test",
+ SampleID: "s2",
+ QAResults: []QAResult{
+ {Category: 1, TokenF1: 0.4, HitRate: 0.6},
+ {Category: 1, TokenF1: 0.6, HitRate: 0.8},
+ },
+ },
+ }
+ for i := range results {
+ results[i].Agg = aggregateMetrics(results[i].QAResults)
+ }
+
+ got := computeModeAgg(results)
+
+ // s1: ValidF1Count=1, F1=0.8; s2: ValidF1Count=2, F1=0.5
+ // Weighted: (0.8*1 + 0.5*2) / 3 = 1.8/3 = 0.6
+ wantF1 := 0.6
+ if math.Abs(got.OverallF1-wantF1) > 1e-9 {
+ t.Errorf("OverallF1 = %.6f, want %.6f", got.OverallF1, wantF1)
+ }
+ if got.ValidF1Count != 3 {
+ t.Errorf("ValidF1Count = %d, want 3", got.ValidF1Count)
+ }
+ if got.TotalQuestions != 4 {
+ t.Errorf("TotalQuestions = %d, want 4", got.TotalQuestions)
+ }
+}
diff --git a/cmd/membench/llm_client.go b/cmd/membench/llm_client.go
new file mode 100644
index 000000000..6c62424da
--- /dev/null
+++ b/cmd/membench/llm_client.go
@@ -0,0 +1,198 @@
+package main
+
+import (
+ "bytes"
+ "context"
+ "encoding/json"
+ "fmt"
+ "io"
+ "log"
+ "net/http"
+ "strings"
+ "time"
+)
+
+// LLMClient wraps an OpenAI-compatible chat completion endpoint.
+type LLMClient struct {
+ BaseURL string
+ Model string
+ APIKey string
+ NoThinking bool // send chat_template_kwargs to disable thinking (llama.cpp specific)
+ MaxRetries int // max retry attempts for transient errors (0 = no retry)
+ Client *http.Client
+}
+
+// LLMClientOptions configures the LLM client.
+type LLMClientOptions struct {
+ BaseURL string
+ Model string
+ APIKey string
+ Timeout time.Duration
+ NoThinking bool
+ MaxRetries int // max retry attempts (default 3)
+}
+
+// NewLLMClient creates a client for an OpenAI-compatible chat completion API.
+func NewLLMClient(opts LLMClientOptions) *LLMClient {
+ if opts.Timeout == 0 {
+ opts.Timeout = 120 * time.Second
+ }
+ maxRetries := opts.MaxRetries
+ if maxRetries < 0 {
+ maxRetries = 3
+ }
+ return &LLMClient{
+ BaseURL: strings.TrimRight(opts.BaseURL, "/"),
+ Model: opts.Model,
+ APIKey: opts.APIKey,
+ NoThinking: opts.NoThinking,
+ MaxRetries: maxRetries,
+ Client: &http.Client{
+ Timeout: opts.Timeout,
+ },
+ }
+}
+
+type chatRequest struct {
+ Model string `json:"model"`
+ Messages []chatMessage `json:"messages"`
+ Temperature float64 `json:"temperature"`
+ MaxTokens int `json:"max_tokens"`
+ ChatTemplateKwargs map[string]any `json:"chat_template_kwargs,omitempty"` // llama.cpp
+ Think *bool `json:"think,omitempty"` // Ollama
+ Thinking map[string]any `json:"thinking,omitempty"` // GLM (智谱)
+}
+
+type chatMessage struct {
+ Role string `json:"role"`
+ Content string `json:"content"`
+}
+
+type chatResponse struct {
+ Choices []struct {
+ Message struct {
+ Content string `json:"content"`
+ ReasoningContent string `json:"reasoning_content,omitempty"`
+ } `json:"message"`
+ } `json:"choices"`
+}
+
+// Complete sends a chat completion request and returns the assistant's reply.
+func (c *LLMClient) Complete(ctx context.Context, systemPrompt, userPrompt string) (string, error) {
+ sysContent := systemPrompt
+ if c.NoThinking && sysContent != "" {
+ // Prepend /no_think tag — works with Ollama /v1 endpoint and
+ // Qwen chat templates where the JSON think field is ignored.
+ sysContent = "/no_think\n" + sysContent
+ }
+ messages := []chatMessage{}
+ if sysContent != "" {
+ messages = append(messages, chatMessage{Role: "system", Content: sysContent})
+ }
+ messages = append(messages, chatMessage{Role: "user", Content: userPrompt})
+
+ body := chatRequest{
+ Model: c.Model,
+ Messages: messages,
+ Temperature: 0.1,
+ MaxTokens: 512,
+ }
+ if c.NoThinking {
+ // llama.cpp: chat_template_kwargs
+ body.ChatTemplateKwargs = map[string]any{
+ "enable_thinking": false,
+ }
+ // Ollama (0.9+): think field
+ thinkFalse := false
+ body.Think = &thinkFalse
+ // GLM (智谱): thinking field
+ body.Thinking = map[string]any{
+ "type": "disabled",
+ }
+ }
+
+ jsonBody, err := json.Marshal(body)
+ if err != nil {
+ return "", fmt.Errorf("marshal request: %w", err)
+ }
+
+ endpoint := strings.TrimRight(c.BaseURL, "/") + "/chat/completions"
+ req, err := http.NewRequestWithContext(ctx, "POST", endpoint, bytes.NewReader(jsonBody))
+ if err != nil {
+ return "", fmt.Errorf("create request: %w", err)
+ }
+ req.Header.Set("Content-Type", "application/json")
+ if c.APIKey != "" {
+ req.Header.Set("Authorization", "Bearer "+c.APIKey)
+ }
+
+ var respBody []byte
+ var lastErr error
+ for attempt := 0; attempt <= c.MaxRetries; attempt++ {
+ if attempt > 0 {
+ backoff := time.Duration(1<<(attempt-1)) * time.Second // 1s, 2s, 4s, ...
+ log.Printf("LLM retry %d/%d after %v: %v", attempt, c.MaxRetries, backoff, lastErr)
+ select {
+ case <-ctx.Done():
+ return "", ctx.Err()
+ case <-time.After(backoff):
+ }
+ // Rebuild request (body reader is consumed)
+ req, err = http.NewRequestWithContext(ctx, "POST", endpoint, bytes.NewReader(jsonBody))
+ if err != nil {
+ return "", fmt.Errorf("create request: %w", err)
+ }
+ req.Header.Set("Content-Type", "application/json")
+ if c.APIKey != "" {
+ req.Header.Set("Authorization", "Bearer "+c.APIKey)
+ }
+ }
+
+ var resp *http.Response
+ resp, lastErr = c.Client.Do(req)
+ if lastErr != nil {
+ continue // network/timeout error → retry
+ }
+
+ respBody, lastErr = io.ReadAll(resp.Body)
+ resp.Body.Close()
+ if lastErr != nil {
+ continue
+ }
+
+ if resp.StatusCode == 429 || resp.StatusCode >= 500 {
+ lastErr = fmt.Errorf("API error %d: %s", resp.StatusCode, string(respBody))
+ continue // rate limit or server error → retry
+ }
+ if resp.StatusCode != 200 {
+ return "", fmt.Errorf("API error %d: %s", resp.StatusCode, string(respBody))
+ }
+
+ lastErr = nil
+ break
+ }
+ if lastErr != nil {
+ return "", fmt.Errorf("after %d retries: %w", c.MaxRetries, lastErr)
+ }
+
+ var chatResp chatResponse
+ if err := json.Unmarshal(respBody, &chatResp); err != nil {
+ return "", fmt.Errorf("parse response: %w", err)
+ }
+ if len(chatResp.Choices) == 0 {
+ return "", fmt.Errorf("no choices in response")
+ }
+ content := strings.TrimSpace(chatResp.Choices[0].Message.Content)
+ // Strip any residual 
+
+
