A JPEG, in pieces, over UDP.
Why Companion fragments still frames across UDP datagrams — and why a dropped frame is, for ambient vision, the right tradeoff.
Companion needs eyes, occasionally. Every so often, the bedside camera grabs a still frame so the conversational model has visual context — a tray on the overbed table, a closed blind, the resident waving — and can speak to what is in the room instead of guessing. This is not video, and it is not surveillance. It is ambient context, sampled at the pace of conversation.
A JPEG, even a small one, is comfortably larger than a single UDP datagram. So the firmware fragments it. Each fragment carries a 12-byte big-endian header: a 4-byte frame id, a 2-byte total fragment count, a 2-byte fragment index, and a 4-byte payload length. The Go API keys an in-memory reassembly buffer on frame id and stitches the JPEG back together as fragments arrive.
Why UDP, and why best-effort
TCP would have given us reliability we do not need and head-of-line blocking we cannot afford. If one fragment of frame N is stuck behind a retransmit, frame N+1 is already stale. UDP lets each frame stand on its own. If a fragment never arrives within a short reassembly window, we drop the partial frame on the floor and move on. No retry, no retransmit storm, no buffer pressure on a device with a finite heap.
The product shape makes this safe. A frame goes to Claude Vision for a one-line description, which is then injected as a system prompt update into the live ElevenLabs ConvAI session. The next opportunity to look is seconds away. A missed frame is a missed glance, not a missed event.
What this means at the bedside
The resident is seen in the way a thoughtful aide notices the room — the water cup is empty, the call light is on, the curtain is drawn. The device does not stream, does not record, does not stare. It glances, occasionally, and forgets. When the conversation needs context, it has some; when it does not, the camera is quiet.