Pinning FreeRTOS tasks across both cores on the ESP32-S3.
How we split network and audio across two cores so a 87-year-old talking to Companion never hears a crackle, stutter, or mid-word pause.
The audio path on the ESP32-S3 inside Companion is a soft real-time problem. The I2S codec hands us a frame every few milliseconds, and we have to keep handing it back on time. If the WiFi stack interrupts the codec task for thirty milliseconds, the user hears a glitch. The user is eighty-seven, sitting alone in a bed they did not choose, talking to a device they are trying to trust. A crackle is not a bug. It is a reason to stop trusting.
Two cores, two jobs.
FreeRTOS on ESP-IDF gives us a preemptive scheduler per core with priorities and affinity. A task with no affinity will be migrated wherever the scheduler thinks is cheapest, which is fine until WiFi housekeeping or an MDNS burst lands on top of the codec. So we pin. Core 0 owns the network world: TCP/IP, MDNS, the WebSocket pump that ships audio frames and JSON commands to the cloud. Core 1 owns the senses: the I2S microphone task, the I2S speaker task, the camera frame grabber.
Priorities are ordered so the things the resident hears win. Mic capture sits at 4, speaker playback at 5, the WebSocket pump at 5 on its own core, the command handler at 3. The speaker outranks the mic because a starved DAC is audible; a one-frame-late mic is not.
Stack tuning is a debugging tool.
We size each task's stack tight and watch the high-water mark with uxTaskGetStackHighWaterMark. Tight stacks catch real bugs early: a recursive JSON parser, a logging format string that pulled in printf floats, a TLS handshake that briefly needs more than we budgeted. A crash on the bench at 2 a.m. is a gift. A reboot in a resident's room at 2 a.m. is not.
What the resident actually hears is the point of all of this. A voice that does not crackle when the cloud reconnects. A reply that does not stutter while the camera grabs a frame. A pause that ends where the sentence ends, not mid-word. The scheduler does its job so Companion can do hers.