Blog
Engineering deep dives, product decisions, and field notes from building Sevah's companion and staff-reviewed workflow for assisted living.
90 posts·4 categories·last updated Jun 1, 2026
Whisper-family models were built to transcribe whole utterances. Force them to stream and you trade away the future context they need to be accurate. At the bedside, where the last word often changes the first, that tradeoff is sharper than it looks.
The features that break autonomous delivery — slow, structured, low manipulation, high human-context — are the features that make care a tractable first deployment.
Task bots have a finish line — book the flight, set the timer. Open-ended companionship has no success signal and nothing to complete, which is exactly why it's so much harder to get right.
Bedside care doesn't yield to clever engineering alone. It needs people who've sat with the problem long before they knew it was one.
TTS leaderboards optimize for naturalness — how human it sounds. For a hard-of-hearing 88-year-old, the most natural voice is often the least intelligible. We tune for comprehension instead.
An annotated timeline of one resident's day, told from Companion's side of the room — from morning check-in through overnight watch.
Drug names are long, rare, foreign-rooted, and full of near-homophones that general speech-to-text was never trained to tell apart. A single wrong med name in a SOAP note is not a typo — it's a patient-safety event.
The device sits with someone between visits. Every other name we tried described a machine doing a job. Companion describes the job.
Field notes from voice sessions with older adults. Competence and warmth in small doses, not a chatty friend or a robot.
Shadow-mode evaluation: a candidate provider or model processes real production conversations in parallel with the live one, and we compare outputs at scale — with zero resident impact, because only the incumbent ever reaches the speaker.
A bench is not a bedroom. Notes from putting CareOS hardware in real rooms: a backlight that became a nightlight, a speaker turned to mud, one bar of cell.
Conversational repair: detecting when Companion misheard a resident and recovering gracefully, without falling into the 'sorry, can you say that again' death spiral.
TTS engines mangle names and drug names confidently and constantly. Custom lexicons and phoneme overrides are how Companion gets Mrs. Okafor — and metoprolol — right every time.
A founder note on the Sanskrit word sevā, what it means precisely, and the discipline a name like this enforces on the product.
TVs, bed alarms, overhead pages, roommates, and hallway carts make a real ward a punishing acoustic environment. Clean-room ASR benchmarks don't survive contact with it. The fix lives on-device and on the server, not in either alone.
A good family update is short, specific, and honest. Companion's daily summary tells you how the night went and what the nurse already did.
Bias does not stop at the microphone. When a model turns a conversation into a SOAP note, it makes editorial choices about what to foreground and what to soften — and the literature is clear about whose symptoms get downplayed. That is why a nurse reviews every note, and why every line is traceable.
Survey week is when a facility learns what wasn't written down. Companion treats documentation as a side effect, not a separate task.
WER is the metric everyone quotes and the one that predicts bedside quality the least. The numbers that actually correlate with a resident being helped are entity error rate, latency p95, turn accuracy, and escalation recall.
Pendants and sensors call after the fall. Companion is built to notice the week before — the gait drift, the medication signs, the risky hour.
Residents don't hold still. They decline, they recover, they come back from a hospital stay as someone new. A cohort assignment is a snapshot of a moving target — so we monitor for drift instead of trusting the original label.
Nurses round on the hour. The resident lives in the gap. Companion makes what happened at 8:47 visible to the nurse who walks in at 10am.
Calling a tool — record a symptom, page a nurse — in the middle of a spoken exchange, without the awkward silence that exposes the machinery underneath.
On-device TTS would mean zero latency and zero network dependence. It would also mean a robot voice in a hospital room. Why Companion synthesizes in the cloud, and what we keep on the chip anyway.
Loneliness shapes health outcomes as much as smoking, yet it almost never makes it onto a care plan. There's no cuff for it. Companion is at the bedside.
Multilingual residents code-switch mid-sentence, and dementia often peels back the layers to a first language learned decades ago. Locking ASR to one language ID is how you build a device that slowly stops understanding someone.
A reminder to drink water every two hours becomes wallpaper. Companion picks the moment, the wording, and sometimes the offer.
A fall is the visible event. The small changes in how a resident moves — days earlier — are the warning Companion is built to catch.
Open speech corpora are full of fluent younger adults reading clean sentences. They contain almost nothing that sounds like a 91-year-old with post-stroke speech in a noisy nursing wing. You cannot model a distribution you never sampled — so we go and sample it.
Audio fixtures, golden transcripts, and tolerance thresholds for a voice pipeline that is flaky by nature — because ASR is non-deterministic and a strict golden test would fail on every green build.
A nurse charting from memory at 6pm loses most of what happened at 11am. Companion captures observations as they occur and drafts the note with provenance.
Cohorts aren't fixed to a person — they're fixed to a person and a time. The resident who is oriented and chatty at 10am is agitated and disoriented at sunset. Time-of-day-aware behavior for sundowning.
Once you generate structured clinical data, the pull is to become the system of record. We chose the opposite. CareOS routes into the EHR a facility already trusts.
Elders process speech more slowly and pause longer mid-thought. Tuning Companion's pacing — silence windows, speaking rate, and prosody — is most of what makes it feel kind.
Companion has a camera and a microphone. By design, it keeps structured observations — not a feed. Here is the technical line and why we drew it.
Humans backchannel, overlap, and listen while they talk. Full-duplex turn-taking is where voice AI is headed — here's what the realtime providers hand us today and what stays genuinely hard.
Partial transcripts that keep rewriting themselves make a device feel uncertain and untrustworthy. How we kill the flicker — debouncing, a stability threshold of N unchanged updates, and committing only the prefix that has stopped moving.
Presbycusis eats consonants first. Our default voice settings are shaped around that fact, not around a generic listener.
Voice cloning is good enough now to recreate a family member from a few minutes of audio. In a memory-care room that is not a feature — it's a deception. Where we draw the line, and why.
Every Companion ships with an embedded LTE modem. No captive portals, no VLAN tickets, no IT change windows. The device boots online and stays online.
Weak, breathy, hypophonic voices arrive at the bedside mic as low-SNR whispers. Crank the gain and you amplify the room; leave it low and you lose the resident. The tradeoff has no free lunch.
A single aggregate WER can look healthy while the device quietly fails an entire cohort. The only honest way to find bias is to stop averaging and start slicing — by accent, age, gender, and clinical condition — and to read the worst slice, not the mean.
One wing, 30 days, per-room pricing, cellular preconfigured. Designed to skip the procurement committee entirely.
LLM-as-judge scales conversation evals to thousands of turns no human could read. It also brings length bias, self-preference, and confident wrongness — so the judge gets calibrated against nurses before we trust a single score.
Alert fatigue kills monitoring products before they earn trust. Companion is built on the opposite premise — silence is the default state.
Post-stroke aphasia breaks the two assumptions every voice stack is built on: that words come out cleanly, and that silence means done talking. Tuning STT and turn detection for the resident who is still searching for the word.
Companion drafts the note. The nurse signs it. The auto-post demo closes deals, but the medical record is a legal artifact — and we drew the line on purpose.
Grounding and hallucination control when Companion is asked about medications and symptoms — and why the right answer is often a refusal that escalates to a human.
The call button assumes four things in a row, and a meaningful slice of residents cannot complete the chain. Companion is built to notice first.
For our residents, the difference between a thinking pause and the end of a turn is the whole problem. Interrupting a word search is the cruelest failure Companion can commit — so we built around it.
The daughter calls the front desk every day at 4pm. She gets voicemail. Designing CareOS for the family member, not the buyer.
Streaming partial transcripts let Companion react before the resident finishes a sentence — but early partials are guesses the model will revise. How we read them for turn and intent signals without ever acting on text that isn't settled yet.
A rover wheeling a tray of pills is the demo investors lean forward for. We built the stationary device first anyway. Here is the sequencing logic.
Default TTS prosody is tuned for fast, fluent listeners. Getting rate, pauses, and emphasis right for an 88-year-old means reaching for SSML — and discovering how unevenly providers support it.
AW88298 on TX, ES7210 on RX, one I2S bus, DMA in PSRAM. The path from datasheet to a conversation that sounds like a conversation.
Dentures, dry mouth, and missing teeth distort exactly the high-frequency sibilants and fricatives that speech-to-text relies on to tell words apart. It's a mundane, deeply eldercare-specific failure that general ASR was never tuned for.
Gender bias in speech tech is rarely loud. It shows up as a higher error rate on one pitch range, a synthesized voice everyone assumed should be female, and assumptions about who is speaking that the acoustics quietly encode.
Why Companion's voice loop on the CoreS3 is an explicit FSM with guarded transitions, and what that bought us in debugging and resident trust.
Offline replay evals and online live metrics catch different bugs and miss different bugs. The trap is trusting either one alone — especially when realtime audio refuses to replay deterministically.
Companion ships cellular, but on first boot it advertises itself over mDNS so a tech can confirm the right device is in the right room.
The hard-of-hearing cohort doesn't need more volume — they need the consonants back. Why presbycusis breaks intelligibility, and why turning the speaker up makes it worse.
Multi-turn memory for one resident, across a conversation and across days — what we keep verbatim, what we summarize, and what we deliberately throw away.
How CareOS resolves voice, prompt, and provider per-device at session start — Firestore for truth, Redis Ring for speed.
Acoustic endpointing hears a pause; semantic endpointing reads whether the sentence is actually complete. Companion fuses both, and the fusion is where the accuracy comes from.
A `context.Context` rooted at server startup lets us ship CareOS in the middle of the day without cutting a single bedside conversation short.
A small neural VAD is dramatically better than an energy gate at telling a whisper from a window unit. The question is not whether it is better — it is whether it fits in flash, RAM, and a 20ms frame on an ESP32-S3, and what we give up by running it server-side instead.
Speech tops out around 8 kHz of bandwidth. 16 kHz is the honest sample rate — anything higher just burns facility wifi.
Total synthesis time barely matters. Time-to-first-audio dominates whether Companion feels alive or broken. Why we chunk by sentence and start speaking before the reply is done.
Stuttering and disfluency don't just garble the transcript — the silent blocks and repeated sounds wreck turn timing, so a hands-free voice device interrupts the one resident who most needs to be let finish.
Companion writes a crash record to NVS flash on its way down, then uploads it on the next boot. A nurse at 3am should never be a debugger.
Speech models are trained overwhelmingly on younger speakers. Aged voices — tremor, weaker pitch control, slower rate, more breathiness — fall outside that distribution, which means the demographic Companion exists for is the one the underlying tech serves worst.
Why CareOS uses a half-duplex gate instead of software AEC on Companion — and why turn-based feels right at the bedside.
Building a labeled conversation eval set from pilot data — consent, de-identification, and the uncomfortable truth that the cases worth grading are the rare, hard ones you have to go hunting for.
Why CareOS runs voice activity detection on the CoreS3 itself — and how a simple RMS gate keeps a bedside conversation feeling like a conversation.
For a resident with dementia, the right answer to a question asked nine times is the same warm answer, nine times. Building a conversation policy that doesn't correct, doesn't argue, and doesn't get tired.
Barge-in sounds simple — stop talking when the resident starts. In practice it means cancelling a reply mid-word, flushing buffers across three machines, and not mistaking your own speaker for the resident.
Why Companion fragments still frames across UDP datagrams — and why a dropped frame is, for ambient vision, the right tradeoff.
Endpointing — deciding the resident has finished their turn — has exactly two ways to fail, and both are felt at the bedside. Here is how we frame the problem and what we measure.
Why CareOS uses a lock-free single-producer single-consumer ring between the mic task and the WebSocket pump — and what that means at the bedside.
A fixed energy threshold is the right place to start and the wrong place to stay. Why a single RMS number fails across rooms and residents, and what spectral features and an adaptive noise floor buy us in the next iteration of on-device VAD.
Cloud realtime, neural vocoders, on-device synthesis — the TTS landscape is wide and none of it was built for a hospital room. How we evaluate and choose per facility.
Why every Companion waits a little longer between reconnect attempts — and why none of them ever give up.
Dysarthria — slurred, weak, imprecise speech from stroke, Parkinson's, or ALS — is common in skilled nursing and brutal for speech-to-text. The acoustics are real signal; the model just wasn't trained to receive them.
Speech-to-text word error rates climb sharply for non-mainstream accents — and in a skilled-nursing wing full of immigrant elders, that gap compounds into a device that quietly stops working for the people who need it most.
Why Companion firmware speaks only OpenAI Realtime — and how the server quietly swaps the model behind it.
You can't tune what you can't measure, and at the bedside the thing you're tuning is a conversation with someone who may not be able to repeat themselves. So the harness comes first.
How we split network and audio across two cores so a 87-year-old talking to Companion never hears a crackle, stutter, or mid-word pause.
Tune Companion for the mean resident and you build a device that fits almost no one. The population isn't a bell curve — it's several distinct peaks. Cohorts, not averages, are the unit of design.
Where the milliseconds actually go between a resident finishing a sentence and Companion starting to talk — and why crossing roughly 800ms makes the whole thing feel broken.
Why CareOS validates every device request with HMAC and a shared secret instead of a database round trip — and what that means at the bedside.