CommunicationPet: Giving the Pet a Body

WhisperaPet could already listen, think and speak, but it had always been an energy orb. This upgrade gives it a body: a character that stands on the desktop, moves its mouth when it talks, and performs actions on request.

Division of labour: two brains, one kept

The body comes from an open-source project called HAST-SEC, which provides VRM rendering, 53-bone IK solving, 26 motion clips, 13 expressions and 5 viseme channels.

• WhisperaPet handles ears, mouth and brain: VAD, ASR, DeepSeek, and the voices and personas in GPT-SoVITS.
• HAST-SEC handles body and face: rendering, bones, motion, expressions.

HAST-SEC ships a 0.8B text model for parsing motion intent, and after the integration it no longer participates — motion intent is produced by the DeepSeek that was already there. Its GGUF and llama-server (about 2GB) never need downloading; the avatar only needs the three dependency and a VRM model.

Data flow

Once the user finishes a sentence, things split into two parallel paths:

• Main reply: DeepSeek streams out → split into sentences → GPT-SoVITS synthesis → audio playback.
• Motion intent: a small parallel DeepSeek request → assistant.motion over WebSocket → executed in the avatar window.

The parallelism is deliberate. The motion intent usually comes back before the first TTS audio chunk, so the character starts moving before it opens its mouth, and end-to-end latency does not grow.

What I built

• Lip sync derived from audio: no second speech recogniser. The TTS audio that is already playing gets analysed, five viseme weights come out of the spectrum, and they are pushed to the renderer at roughly 30Hz. This is the only channel that is truly synchronised with speech.
• Three expressions, three drivers: the mouth follows the audio, the expression follows the real speaking state (held while speaking, then about 1.2 seconds of release back to neutral), and body motion follows the seconds in the motion plan. That last one is deliberately not synced to speech — when the user says “raise your right hand”, that is the instruction itself and should happen immediately, not after the character finishes reading its reply.
• The character replaces the energy orb: instead of a separate window it sits as an embedded <webview> on the main window’s background layer, mutually exclusive with the original shader orb and toggled by a button beside the chat panel. The first toggle waits a dozen seconds for the VRM and policies to load; after that switching is instant.
• Embed mode: a “character only” display mode added to HAST-SEC — transparent background, all of its own panels and buttons hidden, its built-in chat and browser speech turned off, so it does nothing but be a body.
• VRM 0.x compatibility: fixed skeleton validation so VRoid-exported 0.x models import properly.

Walls I hit

1. VRM 0.x renames the thumb bones: 0.x calls them thumbProximal, 1.0 calls them thumbMetacarpal. The validation code read the raw JSON from before the rename, so it judged perfectly valid VRoid characters as “incomplete skeleton” and refused to import them.
2. VRoid models have no ARKit expression channels: HAST-SEC originally mapped happy / angry and friends onto ARKit sub-channels (mouth_smile_1, sp_cheek and so on), but almost every VRoid export only carries the standard VRM expression channels, so expressions vanished entirely. Now both paths run side by side: models with sub-channels use them, models with only standard channels get the standard presets.
3. A failed solve freezes the character permanently: the pose solver has a “no feasible solution” failure flag, and the research build simply pauses on it. That is reasonable for a research demo — you get to see it fail — but it is a disaster for a desktop pet: once the main loop stops, the motion track’s clock stops advancing, so the unsolvable pose is never released, every subsequent resume fails immediately, and it is frozen forever. Embed mode hides the UI and the reset button, so there is no way out. It now auto-recovers instead, and only falls back to pausing after more than five consecutive failures.
4. <webview> does not accept percentage sizes: with % or default sizing the guest viewport stays stuck at 150px (the default height of a replaced element) and the character is squashed into a stripe. It has to be set explicitly in window pixels and synced on resize.
5. max_tokens for motion intent was eaten by reasoning tokens: at 240 the body was truncated, the JSON came out malformed and the motion was silently dropped (one sentence measured 457 reasoning tokens). 1536 fixed it.
6. Lip sync dropped to 1Hz when the main window was occluded: Electron throttles background windows by default and the viseme rate followed it down. Disabling backgroundThrottling in two places stabilised it.

Where it stands

The integration works end to end: speech moves the mouth, expressions change, commands trigger motion. What is not finished is motion precision — there are only 26 discrete “direction + height” motion keys and no positional targets like “put your hand on your chest”, so such a request lands on the nearest approximation.

Two other known limits: loading the VRM takes a dozen seconds, and motion messages arriving in that window are dropped; and motion intent is parsed once per user turn, so a long reply will not change expression partway through.