pulsareon/ShowenV2
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(M2.1): AI 语音对话插件骨架 + 模型管理 + HTTP API + config schema 扩展

Browse Source 
这是 M2.1（语音数字生命 V1）的后端骨架实现，不含 Web UI / Flutter / 设备部署。

新增文件:
- src/plugins/ai/mod.rs: AiPlugin 薄层（持有 ChatPipeline Arc，处理消息系统联动）
- src/plugins/ai/backend.rs: AiBackend trait + LocalCliBackend (whisper-cli/llama-cli/piper 子进程)
  + CloudBackend 占位（V1 返回"暂未支持"）
- src/plugins/ai/chat.rs: ChatPipeline 对话管线执行器（HTTP 层 spawn_blocking 直接调用）
- src/plugins/ai/model_manager.rs: ModelManager 模型资产管理
  (清单/下载/切换/删除/配额/档位守门，参照 plugin_repo + version_manager 模式)

修改文件:
- src/core/message.rs: 新增 ChatRequest/ChatResponse/AiModelEvent 消息类型
- src/core/config.rs: AppConfig 新增 character 块（角色元信息+人设）+ ai 块（后端配置）
  均为 #[serde(default)] 向后兼容旧配置
- src/plugins/mod.rs: 注册 ai 模块
- src/plugins/http/mod.rs: HttpState 新增 ai_pipeline + ai_models 字段及注册方法;
  HttpPlugin 新增 set_ai_pipeline/set_ai_models 方法
- src/plugins/http/routes.rs: 新增 6 个 AI 相关路由
  - POST /api/chat/text (文字对话，Web 端主路径)
  - POST /api/chat/audio (语音对话，App 主路径)
  - GET /api/models (模型清单+水位+配额)
  - POST /api/models/download (下载模型，后台线程执行)
  - POST /api/models/switch (切换激活模型)
  - POST /api/models/delete (删除模型，保护当前激活)
- src/main.rs: 注册 AiPlugin，连接 pipeline 到 HttpPlugin

技术决策:
- 对话管线用 spawn_blocking 而非消息系统，满足 HTTP 同步响应需求
- ChatPipeline 用 Arc<Mutex> 共享，HTTP 和 AiPlugin 共用同一实例
- 互斥用 try_lock，忙时返回 409 而非阻塞
- Spike 结论固化: LLM t=2 锁大核、ctx=1024 限死、Qwen2.5-0.5B 默认档

待完成 (后续提交):
- Web 控制端 UI (文字对话 + 角色切换 + 模型管理界面)
- Flutter App (角色页/语音页/模型管理页)
- 设备端部署 llama.cpp/whisper.cpp/piper 二进制 + 模型下载
- 画面联动 (talk/idle 状态切换)
- 测试

注: Windows 开发环境缺 dbus，无法本地 cargo check；待目标机验证。
This commit is contained in:
pulsareonbot
2026-07-04 15:01:00 +08:00
parent b066dd187b
commit a0c4ca2307
10 changed files with 1487 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

213
src/plugins/ai/backend.rs Normal file
View File

@@ -0,0 +1,213 @@
//! AI 后端 trait + 本地命令行实现
//!
//! 抽象 ASR/LLM/TTS 三层，支持本地命令行（LocalCliBackend）和未来云端后端。
//! V1 只实现 local；cloud 配置时返回"暂未支持"错误。
use anyhow::{bail, Result};
use std::path::{Path, PathBuf};
use std::process::Command;
/// AI 后端抽象 trait
pub trait AiBackend: Send {
/// 语音转文字 (ASR)
/// - audio_path: 16kHz mono wav 临时文件
/// - model_path: whisper.cpp 模型 (ggml-tiny.bin 等)
fn asr(&self, audio_path: &Path, model_path: &Path) -> Result<String>;
/// LLM 对话
/// - model_path: llama.cpp GGUF 模型
/// - persona: 角色人设 system prompt
/// - user_message: 本轮用户输入
/// - history: 历史轮次 (role, content)
/// - max_tokens: 最大回复 token (0=默认 128)
fn llm_chat(
&self,
model_path: &Path,
persona: &str,
user_message: &str,
history: &[(String, String)],
max_tokens: u16,
) -> Result<String>;
/// 文字转语音 (TTS)
/// - text: 要合成的文本
/// - model_path: piper 模型目录/文件
/// - output_path: 输出 wav 文件路径
fn tts(&self, text: &str, model_path: &Path, output_path: &Path) -> Result<()>;
/// 后端标识（local / cloud）
fn backend_name(&self) -> &str;
}
/// 本地命令行后端
///
/// 通过子进程调用 whisper-cli / llama-cli / piper 二进制。
/// Spike (2026-07-03) 验证的命令行参数固化于此。
pub struct LocalCliBackend {
/// 工具目录（含 whisper-cli, llama-cli, piper 二进制）
tools_dir: PathBuf,
}
impl LocalCliBackend {
pub fn new(tools_dir: PathBuf) -> Self {
Self { tools_dir }
}
fn whisper_cli(&self) -> PathBuf {
self.tools_dir.join("whisper-cli")
}
fn llama_cli(&self) -> PathBuf {
self.tools_dir.join("llama-cli")
}
fn piper(&self) -> PathBuf {
self.tools_dir.join("piper")
}
}
impl AiBackend for LocalCliBackend {
fn asr(&self, audio_path: &Path, model_path: &Path) -> Result<String> {
// whisper-cli -m <model> -f <audio> -l zh --no-timestamps -otxt -of <tmp>
// 输出到 <tmp>.txt，读取后返回
let tmp_out = audio_path.with_extension("asr.txt");
let output = Command::new(self.whisper_cli())
.arg("-m")
.arg(model_path)
.arg("-f")
.arg(audio_path)
.arg("-l")
.arg("zh")
.arg("--no-timestamps")
.arg("-otxt")
.arg("-of")
.arg(&tmp_out)
.output()?;
if !output.status.success() {
bail!(
"whisper-cli 失败: {}",
String::from_utf8_lossy(&output.stderr)
);
}
let txt_path = tmp_out.with_extension("txt");
let text = std::fs::read_to_string(&txt_path)?;
let _ = std::fs::remove_file(&txt_path);
Ok(text.trim().to_string())
}
fn llm_chat(
&self,
model_path: &Path,
persona: &str,
user_message: &str,
history: &[(String, String)],
max_tokens: u16,
) -> Result<String> {
// 构造 llama-cli 对话 prompt
// 格式参考 spike: llama-cli -m <model> -p <prompt> -n <max_tokens> -t 2 -c 1024 --temp 0.7
let mut prompt = String::new();
prompt.push_str(&format!("system\n{persona}\n"));
for (role, content) in history {
prompt.push_str(role);
prompt.push('\n');
prompt.push_str(content);
prompt.push('\n');
}
prompt.push_str("user\n");
prompt.push_str(user_message);
prompt.push('\n');
prompt.push_str("assistant\n");
let tokens = if max_tokens == 0 { 128 } else { max_tokens as u32 };
let output = Command::new(self.llama_cli())
.arg("-m")
.arg(model_path)
.arg("-p")
.arg(&prompt)
.arg("-n")
.arg(tokens.to_string())
.arg("-t")
.arg("2") // 锁大核 (spike 结论: t=2 最优)
.arg("-c")
.arg("1024") // 限死上下文 (spike 警告: 默认 ctx 吃 1-3.3G)
.arg("--temp")
.arg("0.7")
.arg("--no-display-prompt")
.output()?;
if !output.status.success() {
bail!(
"llama-cli 失败: {}",
String::from_utf8_lossy(&output.stderr)
);
}
// llama-cli 输出到 stdout，取生成部分
let reply = String::from_utf8_lossy(&output.stdout);
Ok(reply.trim().to_string())
}
fn tts(&self, text: &str, model_path: &Path, output_path: &Path) -> Result<()> {
// echo <text> | piper -m <model> -f <output> --output_format wav
let mut cmd = Command::new(self.piper());
cmd.arg("-m")
.arg(model_path)
.arg("-f")
.arg(output_path)
.arg("--output_format")
.arg("wav")
.stdin(std::process::Stdio::piped())
.stdout(std::process::Stdio::null())
.stderr(std::process::Stdio::piped());
let mut child = cmd.spawn()?;
if let Some(mut stdin) = child.stdin.take() {
use std::io::Write;
stdin.write_all(text.as_bytes())?;
}
let output = child.wait_with_output()?;
if !output.status.success() {
bail!(
"piper 失败: {}",
String::from_utf8_lossy(&output.stderr)
);
}
Ok(())
}
fn backend_name(&self) -> &str {
"local"
}
}
/// 云端后端（V1 仅占位，返回"暂未支持"）
pub struct CloudBackend;
impl AiBackend for CloudBackend {
fn asr(&self, _audio_path: &Path, _model_path: &Path) -> Result<String> {
bail!("云端 ASR 暂未支持 (V1 仅实现 local)")
}
fn llm_chat(
&self,
_model_path: &Path,
_persona: &str,
_user_message: &str,
_history: &[(String, String)],
_max_tokens: u16,
) -> Result<String> {
bail!("云端 LLM 暂未支持 (V1 仅实现 local)")
}
fn tts(&self, _text: &str, _model_path: &Path, _output_path: &Path) -> Result<()> {
bail!("云端 TTS 暂未支持 (V1 仅实现 local)")
}
fn backend_name(&self) -> &str {
"cloud"
}
}

180
src/plugins/ai/chat.rs Normal file
View File

@@ -0,0 +1,180 @@
//! 对话管线执行器（被 HTTP 层直接调用）
//!
//! 为什么不在 AiPlugin 里执行：HTTP 需要同步响应，而 Plugin trait 的
//! handle_message 是异步消息系统，不便返回同步结果。改用 HTTP 层在
//! spawn_blocking 里直接调用此模块，Arc<Mutex> 共享状态。
use crate::core::message::{ChatInput, ChatRequest, ChatResponse};
use crate::plugins::ai::backend::AiBackend;
use crate::plugins::ai::model_manager::ModelManager;
use anyhow::Result;
use std::collections::HashMap;
use std::path::Path;
use std::sync::{Arc, Mutex};
/// 会话上下文
pub struct SessionContext {
pub persona_prompt: String,
pub history: Vec<(String, String)>,
}
/// 对话管线执行器（共享状态）
pub struct ChatPipeline {
pub backend: Mutex<Box<dyn AiBackend>>,
pub models: Arc<Mutex<ModelManager>>,
pub sessions: Mutex<HashMap<String, SessionContext>>,
pub context_window: usize,
pub tmp_dir: std::path::PathBuf,
/// 互斥锁：同一时刻只允许一个回合
pub busy: Mutex<()>,
}
impl ChatPipeline {
pub fn new(
backend: Box<dyn AiBackend>,
models: ModelManager,
tmp_dir: std::path::PathBuf,
context_window: usize,
) -> Arc<Self> {
Arc::new(Self {
backend: Mutex::new(backend),
models: Arc::new(Mutex::new(models)),
sessions: Mutex::new(HashMap::new()),
context_window,
tmp_dir,
busy: Mutex::new(()),
})
}
/// 执行对话回合（同步阻塞，调用方应在 spawn_blocking 里调）
pub fn run(&self, req: &ChatRequest) -> ChatResponse {
// 互斥：拿不到锁说明有进行中的回合
let _guard = match self.busy.try_lock() {
Ok(g) => g,
Err(_) => {
return ChatResponse {
session_id: req.session_id.clone(),
transcription: None,
reply_text: String::new(),
reply_audio_path: None,
error: Some("设备忙，上一个回合未结束".to_string()),
}
}
};
let result = self.run_pipeline(req);
match result {
Ok((transcription, reply_text, audio_path)) => {
// 更新会话上下文
if let Ok(mut sessions) = self.sessions.lock() {
let session = sessions
.entry(req.session_id.clone())
.or_insert_with(|| SessionContext {
persona_prompt: req.persona_prompt.clone(),
history: Vec::new(),
});
if session.persona_prompt != req.persona_prompt {
session.persona_prompt = req.persona_prompt.clone();
session.history.clear();
}
if let Some(t) = &transcription {
session.history.push(("user".to_string(), t.clone()));
}
session.history.push(("assistant".to_string(), reply_text.clone()));
let max = self.context_window * 2;
if session.history.len() > max {
let drain = session.history.len() - max;
session.history.drain(..drain);
}
}
ChatResponse {
session_id: req.session_id.clone(),
transcription,
reply_text,
reply_audio_path: audio_path,
error: None,
}
}
Err(e) => ChatResponse {
session_id: req.session_id.clone(),
transcription: None,
reply_text: String::new(),
reply_audio_path: None,
error: Some(e.to_string()),
},
}
}
fn run_pipeline(&self, req: &ChatRequest) -> Result<(Option<String>, String, Option<String>)> {
// 1. ASR
let transcription = match &req.input {
ChatInput::Text { content } => Some(content.clone()),
ChatInput::Audio { path } => {
let models = self.models.lock().unwrap();
let asr_model = models.active_asr_model_path()?;
drop(models);
let backend = self.backend.lock().unwrap();
Some(backend.asr(Path::new(path), &asr_model)?)
}
};
let user_text = transcription
.as_ref()
.ok_or_else(|| anyhow::anyhow!("ASR 返回空结果"))?;
if user_text.trim().is_empty() {
anyhow::bail!("输入为空（ASR 未识别到语音）");
}
// 2. LLM
let models = self.models.lock().unwrap();
let llm_model = models.active_llm_model_path()?;
drop(models);
let history: Vec<(String, String)> = {
let sessions = self.sessions.lock().unwrap();
sessions
.get(&req.session_id)
.map(|s| s.history.clone())
.unwrap_or_default()
};
let backend = self.backend.lock().unwrap();
let reply_text = backend.llm_chat(
&llm_model,
&req.persona_prompt,
user_text,
&history,
req.max_tokens,
)?;
drop(backend);
// 3. TTS
let models = self.models.lock().unwrap();
let tts_model = models.active_tts_model_path()?;
drop(models);
let audio_path = self.tmp_dir.join(format!(
"tts_{}.wav",
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_millis())
.unwrap_or(0)
));
let backend = self.backend.lock().unwrap();
backend.tts(&reply_text, &tts_model, &audio_path)?;
Ok((
transcription,
reply_text,
Some(audio_path.to_string_lossy().into_owned()),
))
}
/// 清空指定会话上下文（切角色时调用）
pub fn clear_session(&self, session_id: &str) {
if let Ok(mut sessions) = self.sessions.lock() {
sessions.remove(session_id);
}
}
}

161
src/plugins/ai/mod.rs Normal file
View File

@@ -0,0 +1,161 @@
//! AiPlugin — AI 语音对话插件 (M2.1)
//!
//! 设备本地跑 ASR (whisper.cpp) → LLM (llama.cpp) → TTS (piper) 管线，
//! 提供语音/文字对话回合能力。声音一律返回客户端播放，设备不出声。
//!
//! # 架构
//! ```text
//! HTTP /api/chat → ChatPipeline.run() → ASR → LLM → TTS → ChatResponse
//! (Arc<Mutex> 共享, HTTP 在 spawn_blocking 调用)
//! ```
//!
//! AiPlugin 本身是薄层：持有 ChatPipeline Arc 供 HTTP 层取用，
//! 并处理消息系统触发的联动（如 talk/idle 状态切换）。
//!
//! # 硬件约束 (Spike 2026-07-03 张明远实测)
//! - 测试机全志 A733: 2×A78@2.0G + 6×A55@1.8G, 4G 内存
//! - LLM 推理必须 t=2 锁大核 (全核反而崩且挤死视频)
//! - 严禁默认上下文长度 (4096/8192 会吃 1-3.3GB)
//! - 推荐默认 Qwen2.5-0.5B Q4_K_M (16.4 t/s, RSS 985M)
pub mod backend;
pub mod chat;
pub mod model_manager;
pub use chat::{ChatPipeline, SessionContext};
use crate::core::message::{Message, StateChanged};
use crate::core::plugin::*;
use anyhow::Result;
use std::path::PathBuf;
use std::sync::{Arc, Mutex};
/// AiPlugin — AI 对话插件（薄层）
///
/// 持有 ChatPipeline 的共享句柄，供 HTTP 层取用。
/// 自身处理消息系统触发的联动（talk/idle 状态切换等）。
pub struct AiPlugin {
ctx: Option<PluginContext>,
/// 对话管线（HTTP 层通过 Arc clone 直接调用）
pipeline: Arc<ChatPipeline>,
/// 工具目录
tools_dir: PathBuf,
}
impl AiPlugin {
/// 创建默认本地后端实例
pub fn new_default(
model_store: PathBuf,
tools_dir: PathBuf,
tmp_dir: PathBuf,
context_window: usize,
) -> Self {
let backend = Box::new(backend::LocalCliBackend::new(tools_dir.clone()));
let models = model_manager::ModelManager::new(model_store);
let pipeline = ChatPipeline::new(backend, models, tmp_dir, context_window);
Self {
ctx: None,
pipeline,
tools_dir,
}
}
/// 获取对话管线共享句柄（main.rs 注册时传给 HttpPlugin）
pub fn pipeline(&self) -> Arc<ChatPipeline> {
Arc::clone(&self.pipeline)
}
/// 获取模型管理器共享句柄（main.rs 注册时传给 HttpPlugin）
pub fn models(&self) -> Arc<Mutex<model_manager::ModelManager>> {
Arc::clone(&self.pipeline.models)
}
}
impl Plugin for AiPlugin {
fn id(&self) -> &str {
"ai"
}
fn info(&self) -> PluginInfo {
PluginInfo {
name: "AiPlugin".to_string(),
version: "0.1.0".to_string(),
description: "AI 语音对话 (ASR/LLM/TTS 本地推理)".to_string(),
platform: Platform::LinuxArm64,
}
}
fn capabilities(&self) -> Vec<String> {
vec!["chat".to_string(), "model_management".to_string()]
}
fn init(&mut self, ctx: PluginContext) -> Result<()> {
self.ctx = Some(ctx);
Ok(())
}
fn start(&mut self) -> Result<()> {
// 确保临时目录存在
if let Some(tmp) = self.pipeline.tmp_dir.parent() {
std::fs::create_dir_all(tmp).ok();
}
std::fs::create_dir_all(&self.pipeline.tmp_dir).ok();
// 预加载默认模型清单
let mut models = self.pipeline.models.lock().unwrap();
if let Err(e) = models.ensure_default_models() {
eprintln!("[AiPlugin] 警告: 模型初始化失败: {e}");
}
drop(models);
println!(
"[AiPlugin] 启动 (tools={}, tmp={})",
self.tools_dir.display(),
self.pipeline.tmp_dir.display()
);
Ok(())
}
fn handle_message(&mut self, msg: Message) -> Result<()> {
match msg {
Message::ChatRequest(req) => {
// 通过消息系统发起的对话（非 HTTP 路径），同步执行并广播结果
let resp = self.pipeline.run(&req);
if let Some(ctx) = &self.ctx {
let _ = ctx.tx.send(crate::core::message::Envelope {
from: self.id().to_string(),
to: crate::core::message::Destination::Broadcast,
message: Message::ChatResponse(resp),
});
}
}
Message::Shutdown => {
self.stop()?;
}
Message::StateChanged { old_state, new_state } => {
// 画面联动：进入/离开 talk 状态时记录日志（实际联动由 video 插件处理状态机）
if new_state == "talk" || old_state == "talk" {
println!("[AiPlugin] 画面状态: {old_state} → {new_state}");
}
let _ = StateChanged { old_state, new_state }; // 抑制未用警告
}
_ => {}
}
Ok(())
}
fn stop(&mut self) -> Result<()> {
// 清理临时文件
if self.pipeline.tmp_dir.exists() {
if let Ok(entries) = std::fs::read_dir(&self.pipeline.tmp_dir) {
for entry in entries.flatten() {
let path = entry.path();
if path.extension().and_then(|s| s.to_str()) == Some("wav") {
let _ = std::fs::remove_file(&path);
}
}
}
}
Ok(())
}
}

409
src/plugins/ai/model_manager.rs Normal file
View File

@@ -0,0 +1,409 @@
//! ModelManager — AI 模型资产管理 (M2.1)
//!
//! 管理 LLM/ASR/TTS 模型文件，支持清单查看、下载、切换、删除。
//! 参照项目已有 plugin_repo + version_manager 模式设计。
//!
//! # 模型存储结构
//! ```text
//! model_store/
//! ├── registry.json # 本地模型注册表（已下载的模型清单 + active 标记）
//! ├── llm/
//! │ ├── qwen2.5-0.5b-q4_k_m.gguf
//! │ └── gemma3-1b-q4_k_m.gguf
//! ├── asr/
//! │ └── ggml-tiny.bin
//! └── tts/
//! └── zh_CN-huayan-medium.onnx
//! ```
use anyhow::{bail, Context, Result};
use serde::{Deserialize, Serialize};
use std::collections::HashMap;
use std::fs;
use std::path::{Path, PathBuf};
/// 模型类型
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, Hash)]
#[serde(rename_all = "lowercase")]
pub enum ModelKind {
Llm,
Asr,
Tts,
}
impl ModelKind {
pub fn dir_name(&self) -> &'static str {
match self {
ModelKind::Llm => "llm",
ModelKind::Asr => "asr",
ModelKind::Tts => "tts",
}
}
}
/// 模型元信息（仓库清单 + 本地注册表共用）
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ModelInfo {
/// 模型 ID (如 "qwen2.5-0.5b-q4_k_m")
pub id: String,
/// 模型类型
pub kind: ModelKind,
/// 显示名称 (如 "Qwen2.5 0.5B Q4")
pub name: String,
/// 版本
pub version: String,
/// 文件尺寸 (字节)
pub size: u64,
/// 运行内存需求 (字节)
pub memory_required: u64,
/// 推荐设备档位 (如 "4G 内存档可用")
pub recommended_tier: String,
/// 下载 URL（国内镜像源）
pub url: String,
/// SHA256 校验和
pub sha256: String,
/// 文件名（存储到本地的文件名）
pub filename: String,
}
/// 本地注册表条目
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct LocalModelEntry {
#[serde(flatten)]
pub info: ModelInfo,
/// 是否已下载
pub downloaded: bool,
/// 下载进度 (0-100，100=完成)
pub download_progress: u8,
}
/// 本地注册表
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
pub struct ModelRegistry {
/// 所有已知模型（仓库清单 + 本地状态）
pub models: Vec<LocalModelEntry>,
/// 各类型当前激活的模型 ID
pub active: HashMap<ModelKind, String>,
}
/// 模型管理器
pub struct ModelManager {
/// 模型存储根目录
store_dir: PathBuf,
/// 注册表（内存缓存，持久化到 registry.json）
registry: ModelRegistry,
/// 存储配额上限（字节，默认 4GB）
quota: u64,
}
impl ModelManager {
pub fn new(store_dir: PathBuf) -> Self {
let quota = 4 * 1024 * 1024 * 1024; // 4GB
let mut mgr = Self {
store_dir,
registry: ModelRegistry::default(),
quota,
};
let _ = mgr.load_registry();
mgr
}
/// 存储目录
pub fn store_dir(&self) -> &Path {
&self.store_dir
}
/// 配额
pub fn quota(&self) -> u64 {
self.quota
}
/// 已用空间
pub fn used_space(&self) -> u64 {
self.registry
.models
.iter()
.filter(|m| m.downloaded)
.map(|m| m.info.size)
.sum()
}
/// 加载本地注册表
fn load_registry(&mut self) -> Result<()> {
let path = self.store_dir.join("registry.json");
if path.exists() {
let content = fs::read_to_string(&path)?;
self.registry = serde_json::from_str(&content)?;
}
Ok(())
}
/// 持久化注册表
fn save_registry(&self) -> Result<()> {
fs::create_dir_all(&self.store_dir)?;
let path = self.store_dir.join("registry.json");
let content = serde_json::to_string_pretty(&self.registry)?;
fs::write(&path, content)?;
Ok(())
}
/// 初始化默认模型清单（首次启动或清单缺失时）
pub fn ensure_default_models(&mut self) -> Result<()> {
if !self.registry.models.is_empty() {
return Ok(());
}
// Spike 推荐的默认模型清单
let defaults = vec![
ModelInfo {
id: "qwen2.5-0.5b-q4_k_m".to_string(),
kind: ModelKind::Llm,
name: "Qwen2.5 0.5B Q4 (推荐默认)".to_string(),
version: "0.5b".to_string(),
size: 469 * 1024 * 1024,
memory_required: 1024 * 1024 * 1024,
recommended_tier: "4G 内存档可用".to_string(),
url: "https://hf-mirror.com/Qwen/Qwen2.5-0.5B-Instruct-GGUF/resolve/main/qwen2.5-0.5b-instruct-q4_k_m.gguf".to_string(),
sha256: String::new(), // 实际部署时填充
filename: "qwen2.5-0.5b-q4_k_m.gguf".to_string(),
},
ModelInfo {
id: "gemma3-1b-q4_k_m".to_string(),
kind: ModelKind::Llm,
name: "Gemma3 1B Q4 (可选档)".to_string(),
version: "1b".to_string(),
size: 800 * 1024 * 1024,
memory_required: 2 * 1024 * 1024 * 1024,
recommended_tier: "建议 8G 以上".to_string(),
url: "https://hf-mirror.com/google/gemma-3-1b-it-GGUF/resolve/main/gemma-3-1b-it-q4_k_m.gguf".to_string(),
sha256: String::new(),
filename: "gemma3-1b-q4_k_m.gguf".to_string(),
},
ModelInfo {
id: "whisper-tiny".to_string(),
kind: ModelKind::Asr,
name: "Whisper Tiny (中文)".to_string(),
version: "tiny".to_string(),
size: 75 * 1024 * 1024,
memory_required: 200 * 1024 * 1024,
recommended_tier: "4G 内存档可用".to_string(),
url: "https://hf-mirror.com/ggerganov/whisper.cpp/resolve/main/ggml-tiny.bin".to_string(),
sha256: String::new(),
filename: "ggml-tiny.bin".to_string(),
},
ModelInfo {
id: "piper-zh-huayan".to_string(),
kind: ModelKind::Tts,
name: "Piper zh_CN huayan (中文女声)".to_string(),
version: "medium".to_string(),
size: 63 * 1024 * 1024,
memory_required: 150 * 1024 * 1024,
recommended_tier: "4G 内存档可用".to_string(),
url: "https://hf-mirror.com/rhasspy/piper-voices/resolve/main/zh/zh_CN/huayan/medium/zh_CN-huayan-medium.onnx".to_string(),
sha256: String::new(),
filename: "zh_CN-huayan-medium.onnx".to_string(),
},
];
for info in defaults {
let filename = info.filename.clone();
let downloaded = self.store_dir.join(info.kind.dir_name()).join(&filename).exists();
self.registry.models.push(LocalModelEntry {
info,
downloaded,
download_progress: if downloaded { 100 } else { 0 },
});
}
// 设置默认 active 模型
if !self.registry.active.contains_key(&ModelKind::Llm) {
self.registry.active.insert(ModelKind::Llm, "qwen2.5-0.5b-q4_k_m".to_string());
}
if !self.registry.active.contains_key(&ModelKind::Asr) {
self.registry.active.insert(ModelKind::Asr, "whisper-tiny".to_string());
}
if !self.registry.active.contains_key(&ModelKind::Tts) {
self.registry.active.insert(ModelKind::Tts, "piper-zh-huayan".to_string());
}
self.save_registry()?;
Ok(())
}
/// 列出所有模型
pub fn list_models(&self) -> &[LocalModelEntry] {
&self.registry.models
}
/// 获取当前激活的模型 ID
pub fn active_model_id(&self, kind: ModelKind) -> Option<&String> {
self.registry.active.get(&kind)
}
/// 获取已激活 LLM 模型文件路径
pub fn active_llm_model_path(&self) -> Result<PathBuf> {
self.active_model_path(ModelKind::Llm)
}
/// 获取已激活 ASR 模型文件路径
pub fn active_asr_model_path(&self) -> Result<PathBuf> {
self.active_model_path(ModelKind::Asr)
}
/// 获取已激活 TTS 模型文件路径
pub fn active_tts_model_path(&self) -> Result<PathBuf> {
self.active_model_path(ModelKind::Tts)
}
fn active_model_path(&self, kind: ModelKind) -> Result<PathBuf> {
let id = self
.registry
.active
.get(&kind)
.ok_or_else(|| anyhow::anyhow!("没有激活的 {kind:?} 模型"))?;
let entry = self
.registry
.models
.iter()
.find(|m| &m.info.id == id)
.ok_or_else(|| anyhow::anyhow!("激活的模型 {id} 不在注册表中"))?;
if !entry.downloaded {
bail!("激活的模型 {id} 尚未下载");
}
Ok(self.store_dir.join(kind.dir_name()).join(&entry.info.filename))
}
/// 下载模型
pub fn download_model(&mut self, model_id: &str) -> Result<()> {
let entry = self
.registry
.models
.iter()
.find(|m| m.info.id == model_id)
.ok_or_else(|| anyhow::anyhow!("模型 {model_id} 不在清单中"))?
.clone();
if entry.downloaded {
return Ok(()); // 已下载
}
// 配额检查
if self.used_space() + entry.info.size > self.quota {
bail!(
"磁盘配额不足: 已用 {} + 需要 {} > 配额 {}",
self.used_space(),
entry.info.size,
self.quota
);
}
// 确保目录存在
let dir = self.store_dir.join(entry.info.kind.dir_name());
fs::create_dir_all(&dir)?;
let dest = dir.join(&entry.info.filename);
// 下载（简单实现，不带断点续传；生产环境应加）
println!("[ModelManager] 下载模型 {} from {}", model_id, entry.info.url);
let resp = ureq::get(&entry.info.url)
.timeout(std::time::Duration::from_secs(600))
.call()
.context("下载模型失败")?;
let mut file = fs::File::create(&dest)?;
std::io::copy(&mut resp.into_reader(), &mut file)?;
// 标记为已下载
if let Some(m) = self.registry.models.iter_mut().find(|m| m.info.id == model_id) {
m.downloaded = true;
m.download_progress = 100;
}
self.save_registry()?;
println!("[ModelManager] 模型 {} 下载完成", model_id);
Ok(())
}
/// 切换激活模型
pub fn switch_model(&mut self, kind: ModelKind, model_id: &str) -> Result<()> {
let entry = self
.registry
.models
.iter()
.find(|m| m.info.id == model_id && m.info.kind == kind)
.ok_or_else(|| anyhow::anyhow!("模型 {model_id} 不在 {kind:?} 清单中"))?;
if !entry.downloaded {
bail!("模型 {model_id} 尚未下载，无法切换");
}
let old = self.registry.active.insert(kind, model_id.to_string());
self.save_registry()?;
if let Some(old_id) = old {
println!("[ModelManager] {kind:?} 模型切换: {old_id} → {model_id}");
} else {
println!("[ModelManager] {kind:?} 模型激活: {model_id}");
}
Ok(())
}
/// 删除模型（当前激活的模型不可删除）
pub fn delete_model(&mut self, model_id: &str) -> Result<()> {
let entry = self
.registry
.models
.iter()
.find(|m| m.info.id == model_id)
.ok_or_else(|| anyhow::anyhow!("模型 {model_id} 不在清单中"))?;
// 保护当前激活模型
if let Some(active_id) = self.registry.active.get(&entry.info.kind) {
if active_id == model_id {
bail!("模型 {model_id} 正在使用中，无法删除（请先切换到其他模型）");
}
}
if !entry.downloaded {
bail!("模型 {model_id} 未下载，无需删除");
}
// 删除文件
let path = self
.store_dir
.join(entry.info.kind.dir_name())
.join(&entry.info.filename);
if path.exists() {
fs::remove_file(&path)?;
}
// 更新注册表
if let Some(m) = self.registry.models.iter_mut().find(|m| m.info.id == model_id) {
m.downloaded = false;
m.download_progress = 0;
}
self.save_registry()?;
println!("[ModelManager] 模型 {} 已删除", model_id);
Ok(())
}
/// 获取注册表（用于 HTTP API 返回）
pub fn registry(&self) -> &ModelRegistry {
&self.registry
}
}
/// 仓库清单索引（远程静态清单文件）
#[derive(Debug, Clone, Deserialize)]
pub struct RepoModelIndex {
pub models: Vec<ModelInfo>,
}
impl ModelManager {
/// 从远程仓库拉取最新清单（更新本地注册表，不触发下载）
pub fn fetch_repo_index(repo_url: &str) -> Result<RepoModelIndex> {
let resp = ureq::get(&format!("{repo_url}/models/index.json"))
.timeout(std::time::Duration::from_secs(30))
.call()
.context("拉取模型清单失败")?;
let index: RepoModelIndex = serde_json::from_reader(resp.into_reader())?;
Ok(index)
}
}

61
src/plugins/http/mod.rs
View File

@@ -48,6 +48,10 @@ pub(crate) struct HttpState {
ws_events: broadcast::Sender<String>,
/// 动态插件管理状态（由 Custom 消息更新）
plugin_states: Mutex<Vec<crate::core::service_manager::PluginStateInfo>>,
/// AI 对话管线（HTTP 路由直接调用）
ai_pipeline: Mutex<Option<std::sync::Arc<crate::plugins::ai::ChatPipeline>>>,
/// AI 模型管理器共享句柄（HTTP 模型管理 API 调用）
ai_models: Mutex<Option<std::sync::Arc<std::sync::Mutex<crate::plugins::ai::model_manager::ModelManager>>>>,
}
impl HttpState {
@@ -75,9 +79,40 @@ impl HttpState {
ble_ready: AtomicBool::new(false),
ws_events,
plugin_states: Mutex::new(Vec::new()),
ai_pipeline: Mutex::new(None),
ai_models: Mutex::new(None),
}
}
/// 注册 AI 对话管线（AiPlugin init 时调用）
pub(crate) fn register_ai_pipeline(&self, pipeline: std::sync::Arc<crate::plugins::ai::ChatPipeline>) {
if let Ok(mut slot) = self.ai_pipeline.lock() {
*slot = Some(pipeline);
}
}
/// 注册 AI 模型管理器（AiPlugin init 时调用）
pub(crate) fn register_ai_models(
&self,
models: std::sync::Arc<std::sync::Mutex<crate::plugins::ai::model_manager::ModelManager>>,
) {
if let Ok(mut slot) = self.ai_models.lock() {
*slot = Some(models);
}
}
/// 获取 AI 对话管线（HTTP 路由调用）
pub(crate) fn ai_pipeline(&self) -> Option<std::sync::Arc<crate::plugins::ai::ChatPipeline>> {
self.ai_pipeline.lock().ok().and_then(|slot| slot.clone())
}
/// 获取 AI 模型管理器（HTTP 路由调用）
pub(crate) fn ai_models(
&self,
) -> Option<std::sync::Arc<std::sync::Mutex<crate::plugins::ai::model_manager::ModelManager>>> {
self.ai_models.lock().ok().and_then(|slot| slot.clone())
}
fn publish_wifi_result(&self, payload: String) {
if let Ok(mut state) = self.wifi_response.lock() {
state.version += 1;
@@ -210,6 +245,10 @@ pub struct HttpPlugin {
state: Option<Arc<HttpState>>,
shutdown_tx: Option<oneshot::Sender<()>>,
server_thread: Option<JoinHandle<()>>,
/// AI 对话管线（main.rs 注册时传入，init 时存入 HttpState）
ai_pipeline: Option<std::sync::Arc<crate::plugins::ai::ChatPipeline>>,
/// AI 模型管理器（main.rs 注册时传入）
ai_models: Option<std::sync::Arc<std::sync::Mutex<crate::plugins::ai::model_manager::ModelManager>>>,
}
impl HttpPlugin {
@@ -219,8 +258,20 @@ impl HttpPlugin {
state: None,
shutdown_tx: None,
server_thread: None,
ai_pipeline: None,
ai_models: None,
}
}
/// 设置 AI 对话管线（main.rs 注册 AiPlugin 后调用）
pub fn set_ai_pipeline(&mut self, pipeline: std::sync::Arc<crate::plugins::ai::ChatPipeline>) {
self.ai_pipeline = Some(pipeline);
}
/// 设置 AI 模型管理器（main.rs 注册 AiPlugin 后调用）
pub fn set_ai_models(&mut self, models: std::sync::Arc<std::sync::Mutex<crate::plugins::ai::model_manager::ModelManager>>) {
self.ai_models = Some(models);
}
}
impl Default for HttpPlugin {
@@ -248,7 +299,15 @@ impl Plugin for HttpPlugin {
}
fn init(&mut self, ctx: PluginContext) -> Result<()> {
self.state = Some(Arc::new(HttpState::new(Arc::clone(&ctx.config))));
let state = Arc::new(HttpState::new(Arc::clone(&ctx.config)));
// 注册 AI 句柄（如果 main.rs 已设置）
if let Some(pipeline) = self.ai_pipeline.take() {
state.register_ai_pipeline(pipeline);
}
if let Some(ai_models) = self.ai_models.take() {
state.register_ai_models(ai_models);
}
self.state = Some(state);
self.ctx = Some(ctx);
Ok(())
}

276
src/plugins/http/routes.rs
View File

@@ -1,7 +1,7 @@
use super::HttpState;
use crate::core::config::{self, AppConfig};
use crate::core::dispatch;
use crate::core::message::{Destination, Envelope, Message, PlayerCommand, WifiCommand};
use crate::core::message::{ChatInput, ChatRequest, Destination, Envelope, Message, PlayerCommand, WifiCommand};
use bytes::Buf;
use futures_util::{SinkExt, StreamExt, TryStreamExt};
use serde::de::DeserializeOwned;
@@ -145,7 +145,15 @@ pub(crate) fn build_routes(
.or(file_mkdir_route(Arc::clone(&state)))
.boxed();
let api = core_api.or(media_api).or(plugin_api).or(file_api);
let ai_api = chat_text_route(Arc::clone(&state))
.or(chat_audio_route(Arc::clone(&state)))
.or(models_list_route(Arc::clone(&state)))
.or(model_download_route(Arc::clone(&state)))
.or(model_switch_route(Arc::clone(&state)))
.or(model_delete_route(Arc::clone(&state)))
.boxed();
let api = core_api.or(media_api).or(plugin_api).or(file_api).or(ai_api);
root_route()
.or(download_route(Arc::clone(&state)))
@@ -1954,6 +1962,270 @@ async fn send_plugin_command(
}
}
// ── AI 对话 API (M2.1) ──
#[derive(Deserialize)]
struct ChatTextRequest {
#[serde(default)]
session_id: Option<String>,
text: String,
}
/// POST /api/chat/text — 文字对话（Web 端主路径）
fn chat_text_route(
state: Arc<HttpState>,
) -> impl Filter<Extract = impl Reply, Error = warp::Rejection> + Clone {
warp::path!("api" / "chat" / "text")
.and(warp::post())
.and(warp::body::json::<ChatTextRequest>())
.and(with_state(state))
.and_then(|req: ChatTextRequest, state: Arc<HttpState>| async move {
let pipeline = match state.ai_pipeline() {
Some(p) => p,
None => {
return Ok::<_, Infallible>(error_json(
StatusCode::SERVICE_UNAVAILABLE,
"AI 插件未就绪",
))
}
};
let config = state.config();
let session_id = req
.session_id
.unwrap_or_else(|| format!("web_{}", std::process::id()));
let chat_req = ChatRequest {
session_id,
input: ChatInput::Text {
content: req.text,
},
persona_prompt: config.character.persona_prompt.clone(),
max_tokens: config.character.max_tokens,
};
// AI 管线是同步阻塞调用（ASR/LLM/TTS 子进程），用 spawn_blocking 避免阻塞 tokio reactor
let resp = tokio::task::spawn_blocking(move || pipeline.run(&chat_req))
.await
.unwrap_or_else(|e| ChatResponse {
session_id: String::new(),
transcription: None,
reply_text: String::new(),
reply_audio_path: None,
error: Some(format!("AI 管线执行失败: {e}")),
});
if resp.error.is_some() {
Ok(json_response(StatusCode::INTERNAL_SERVER_ERROR, &resp))
} else {
Ok(json_response(StatusCode::OK, &resp))
}
})
}
/// POST /api/chat/audio — 语音对话（App 主路径，上传 wav）
fn chat_audio_route(
state: Arc<HttpState>,
) -> impl Filter<Extract = impl Reply, Error = warp::Rejection> + Clone {
warp::path!("api" / "chat" / "audio")
.and(warp::post())
.and(warp::body::content_length_limit(20 * 1024 * 1024))
.and(warp::body::bytes())
.and(with_state(state))
.and_then(|bytes: bytes::Bytes, state: Arc<HttpState>| async move {
let pipeline = match state.ai_pipeline() {
Some(p) => p,
None => {
return Ok::<_, Infallible>(error_json(
StatusCode::SERVICE_UNAVAILABLE,
"AI 插件未就绪",
))
}
};
let config = state.config();
let tmp_dir = config.ai.tmp_dir.clone();
let _ = std::fs::create_dir_all(&tmp_dir);
let audio_path = tmp_dir.join(format!(
"asr_{}.wav",
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_millis())
.unwrap_or(0)
));
if let Err(e) = std::fs::write(&audio_path, &bytes) {
return Ok(error_json(
StatusCode::INTERNAL_SERVER_ERROR,
&format!("保存音频失败: {e}"),
));
}
let session_id = format!("app_{}", std::process::id());
let chat_req = ChatRequest {
session_id,
input: ChatInput::Audio {
path: audio_path.to_string_lossy().into_owned(),
},
persona_prompt: config.character.persona_prompt.clone(),
max_tokens: config.character.max_tokens,
};
let audio_path_clone = audio_path.clone();
let resp = tokio::task::spawn_blocking(move || pipeline.run(&chat_req))
.await
.unwrap_or_else(|e| ChatResponse {
session_id: String::new(),
transcription: None,
reply_text: String::new(),
reply_audio_path: None,
error: Some(format!("AI 管线执行失败: {e}")),
});
// 清理上传的临时音频
let _ = std::fs::remove_file(&audio_path_clone);
if resp.error.is_some() {
Ok(json_response(StatusCode::INTERNAL_SERVER_ERROR, &resp))
} else {
Ok(json_response(StatusCode::OK, &resp))
}
})
}
// ── AI 模型管理 API (M2.1) ──
/// GET /api/models — 列出所有模型
fn models_list_route(
state: Arc<HttpState>,
) -> impl Filter<Extract = impl Reply, Error = warp::Rejection> + Clone {
warp::path!("api" / "models")
.and(warp::get())
.and(with_state(state))
.and_then(|state: Arc<HttpState>| async move {
let models = match state.ai_models() {
Some(m) => m,
None => {
return Ok::<_, Infallible>(error_json(
StatusCode::SERVICE_UNAVAILABLE,
"AI 插件未就绪",
))
}
};
let mgr = models.lock().unwrap();
let registry = mgr.registry();
let used = mgr.used_space();
let quota = mgr.quota();
let result = serde_json::json!({
"models": registry.models,
"active": registry.active,
"used_space": used,
"quota": quota,
});
Ok(json_response(StatusCode::OK, &result))
})
}
#[derive(Deserialize)]
struct ModelActionRequest {
model_id: String,
}
/// POST /api/models/download — 下载模型
fn model_download_route(
state: Arc<HttpState>,
) -> impl Filter<Extract = impl Reply, Error = warp::Rejection> + Clone {
warp::path!("api" / "models" / "download")
.and(warp::post())
.and(warp::body::json::<ModelActionRequest>())
.and(with_state(state))
.and_then(|req: ModelActionRequest, state: Arc<HttpState>| async move {
let models = match state.ai_models() {
Some(m) => m,
None => {
return Ok::<_, Infallible>(error_json(
StatusCode::SERVICE_UNAVAILABLE,
"AI 插件未就绪",
))
}
};
// 下载在独立线程执行，避免阻塞 HTTP
let models_clone = models.clone();
let model_id = req.model_id.clone();
std::thread::spawn(move || {
let mut mgr = models_clone.lock().unwrap();
if let Err(e) = mgr.download_model(&model_id) {
eprintln!("[HttpPlugin] 模型下载失败 {model_id}: {e}");
}
});
Ok::<_, Infallible>(success_json(format!("模型 {} 下载已启动", req.model_id)))
})
}
/// POST /api/models/switch — 切换激活模型
fn model_switch_route(
state: Arc<HttpState>,
) -> impl Filter<Extract = impl Reply, Error = warp::Rejection> + Clone {
warp::path!("api" / "models" / "switch")
.and(warp::post())
.and(warp::body::json::<serde_json::Value>())
.and(with_state(state))
.and_then(|body: serde_json::Value, state: Arc<HttpState>| async move {
let models = match state.ai_models() {
Some(m) => m,
None => {
return Ok::<_, Infallible>(error_json(
StatusCode::SERVICE_UNAVAILABLE,
"AI 插件未就绪",
))
}
};
let model_id = match body.get("model_id").and_then(|v| v.as_str()) {
Some(s) => s.to_string(),
None => return Ok(error_json(StatusCode::BAD_REQUEST, "缺少 model_id")),
};
let kind_str = match body.get("kind").and_then(|v| v.as_str()) {
Some(s) => s,
None => return Ok(error_json(StatusCode::BAD_REQUEST, "缺少 kind")),
};
let kind = match kind_str {
"llm" => crate::plugins::ai::model_manager::ModelKind::Llm,
"asr" => crate::plugins::ai::model_manager::ModelKind::Asr,
"tts" => crate::plugins::ai::model_manager::ModelKind::Tts,
_ => return Ok(error_json(StatusCode::BAD_REQUEST, "kind 必须为 llm/asr/tts")),
};
let mut mgr = models.lock().unwrap();
match mgr.switch_model(kind, &model_id) {
Ok(()) => Ok(success_json(format!("模型已切换为 {}", model_id))),
Err(e) => Ok(error_json(StatusCode::BAD_REQUEST, &e.to_string())),
}
})
}
/// POST /api/models/delete — 删除模型
fn model_delete_route(
state: Arc<HttpState>,
) -> impl Filter<Extract = impl Reply, Error = warp::Rejection> + Clone {
warp::path!("api" / "models" / "delete")
.and(warp::post())
.and(warp::body::json::<ModelActionRequest>())
.and(with_state(state))
.and_then(|req: ModelActionRequest, state: Arc<HttpState>| async move {
let models = match state.ai_models() {
Some(m) => m,
None => {
return Ok::<_, Infallible>(error_json(
StatusCode::SERVICE_UNAVAILABLE,
"AI 插件未就绪",
))
}
};
let mut mgr = models.lock().unwrap();
match mgr.delete_model(&req.model_id) {
Ok(()) => Ok(success_json(format!("模型 {} 已删除", req.model_id))),
Err(e) => Ok(error_json(StatusCode::BAD_REQUEST, &e.to_string())),
}
})
}
fn json_response<T: Serialize>(status: StatusCode, payload: &T) -> warp::reply::Response {
warp::reply::with_status(warp::reply::json(payload), status).into_response()
}

1
src/plugins/mod.rs
View File

@@ -1,3 +1,4 @@
pub mod ai;
pub mod ble;
pub mod device;
pub mod http;