When the person you're talking to is vulnerable, a confident wrong answer is the worst outcome.
Grounding and hallucination control when Companion is asked about medications and symptoms — and why the right answer is often a refusal that escalates to a human.
A resident at 2am: I don't think I took my blood pressure pill, should I take another one? A general-purpose model will happily answer. That is exactly the problem. The fluency that makes Companion feel like company is the same fluency that makes a wrong answer sound trustworthy, and the person on the other end is in no position to second-guess it. For most chatbots a hallucination is embarrassing. Here it can be dangerous.
Grounded, or silent
Our rule is that clinical claims must be grounded in data we actually hold, or not made at all. Companion does not reason about a resident's medication schedule from training-data priors. If the answer requires a fact — what's on the chart, when a dose is due, what a number means — and that fact isn't in the resident's structured record passed into the session, the model is instructed to say so and route to a person rather than fill the gap with a plausible guess.
Concretely, the system prompt draws hard boundaries the model is not permitted to cross on its own:
- No dosing decisions. Whether to take, skip, or double a medication is never Companion's call. That question becomes an escalation, full stop.
- No diagnosis. Is this a heart attack? is not a question to answer; it is a signal to get a nurse, immediately.
- No invented specifics. If Companion doesn't have the chart in front of it, it doesn't quote the chart.
Refusal that doesn't feel like a wall
A blunt I can't help with that would be its own kind of abandonment to someone scared at 2am. So a refusal is never the end of the turn — it's a handoff. Companion acknowledges the worry, says plainly that a nurse should answer this one, and fires a structured escalation event to the floor. The resident hears warmth and a next step, not a dead end. The clinical judgment stays with the human who is licensed to make it.
We treat the boundary as a tripwire, not a suggestion. Med and symptom intents are detected and force the escalation path regardless of how the model would otherwise reply, because a safety rule that depends on the model choosing to follow it isn't a safety rule. And nothing the model produces about a symptom is ever clinical output on its own — it becomes an event that a nurse reviews before it counts for anything.
So the resident in 214B asks her 2am question, and Companion doesn't pretend to be her doctor. It says someone who knows her chart will check on the pill, and it makes sure that someone actually does. The most caring thing a voice in a dark room can do is know the edge of what it knows.