WhisperaPet

The conversation logic was already mine; what was missing was a real-time voice framework that actually runs. WhisperaPet is those two halves joined together.

Approach

Keep Whispera’s real-time voice capability — Electron front end, Silero VAD barge-in, WebSocket protocol, streamed playback — and replace the local model stack it ships with:

• Local llama-server + GGUF → DeepSeek API
• Local VoxCPM streaming TTS → GPT-SoVITS api_v2 service
• ASR from SenseVoiceSmall → the paraformer snapshot bundled with GPT-SoVITS

What I built

• Persona system: a personas/<id>/ directory per persona, each holding a SKILL.md (persona skill) and a voice.json (reference audio, prompt text and optional fine-tuned weights). Adding a persona means adding a directory, not changing code.
• One-switch persona change: switching updates the LLM system prompt, the TTS reference audio and the GPT-SoVITS fine-tuned weights in one step, and the front end confirms once it takes effect.
• Weight caching: GPT-SoVITS weights are global server state, so loaded combinations are cached instead of reloading on every synthesis. Switching back and forth between fine-tuned and zero-shot personas does not bleed voices.
• One-click launcher: wraps Python interpreter reuse, TTS service startup, model validation and the Electron launch into a single script.

Two walls I hit

1. The Python 3.9 type-annotation wall: GPT-SoVITS ships a portable Python 3.9 runtime, but the backend routes used 3.10+ X | Y union annotations, so FastAPI crashed while registering routes.
2. ASR incompatible with the funasr version: the SenseVoiceSmall snapshot lacks the dims metadata funasr 1.0.27 expects and failed to load with 10 missing required arguments. The fix was to reuse the paraformer Chinese snapshot that GPT-SoVITS already bundles and uses daily.

Both are written up with symptoms, root cause and verification.