Six Transport Failures, One Model Migration, and the Route to a Stable Real-Time Voice Agent

Blocking C-Extension Calls Freezing the Event Loop

PyAudio's audio_stream.read() and speaker_stream.write() are synchronous C-extension calls. Running them directly inside an asyncio task froze Python's single thread while waiting for physical soundcard buffer fills or drains. Because the event loop deadlocked during these calls, background WebSocket ping/pong control frames were dropped, prompting the server to drop the transport.

The fix was wrapping both hardware read and write calls inside thread executors:

data = await loop.run_in_executor(None, audio_stream.read, CHUNK_SIZE, False)

# Speaker side — same problem, same fix
await loop.run_in_executor(None, speaker_stream.write, part.inline_data.data)

Even with the executor, the receive loop could still starve between chunks. A yield before each write gave the WebSocket reader a chance to process control frames:

await asyncio.sleep(0)
await loop.run_in_executor(None, speaker_stream.write, part.inline_data.data)

Lesson: never execute blocking hardware C-extension calls directly on an asyncio event loop — on either the input or the output side.

Queue Flooding and Socket Saturation

While the model was speaking for several seconds, PyAudio accumulated raw PCM audio chunks in C memory buffers. When the model finished, the loop drained these buffered frames instantly, flooding the WebSocket with dozens of stale audio frames in milliseconds, exceeding write buffer thresholds.

The fix was decoupling capture from transmission using an asyncio.Queue() and actively flushing hardware input buffers while the model was speaking:

if is_model_speaking:
    audio_stream.read(audio_stream.get_read_available(), False)
    await asyncio.sleep(0.05)

Lesson: clear hardware audio capture buffers during output playback to prevent queue flooding.

Outbound Transport Silence During Playback

To prevent feedback loops, the client ceased sending microphone frames while the model was speaking. However, outbound silence across even short playback windows violated transport activity requirements, causing the server gateway to assume client disconnection and issue error 1011. In practice the drops fired after only 2–3 seconds of send-side quiet, not the multi-second threshold initially assumed.

An intermediate fix sent an explicit audio_stream_end marker followed by zero-padded PCM silence frames (b"\x00" * 3200) every 100ms during playback to keep the transport active. But sending silence after the end marker confused the server-side Voice Activity Detection — the model would sit idle waiting for an utterance that never properly terminated.

The correct pattern was to send audio_stream_end once, then stop sending audio entirely and simply await the model's turn:

if is_model_speaking:
    # Flush stale queue
    while not audio_queue.empty():
        audio_queue.get_nowait()

    if not stream_end_sent:
        await session.send_realtime_input(audio_stream_end=True)
        stream_end_sent = True

    # Do NOT send silence. Sleep and yield to the receive loop.
    await asyncio.sleep(0.02)
    continue

Lesson: signal the end of the audio stream explicitly; do not paper over the pause with silence frames, which the VAD will misinterpret.

Session Resumption Is Vertex-Only — No Built-In Reconnect

Configuring session_resumption=types.SessionResumptionConfig(transparent=True) caused immediate connection rejection: Session Error: transparent parameter is only supported in Gemini Enterprise Agent Platform mode, not in Gemini Developer API mode.

This is not a bug in the SDK — the parameter is legitimate, but it is gated to Vertex AI / Gemini Enterprise. On the public Developer API (the mode used when authenticating with a GOOGLE_API_KEY), there is no built-in session resumption at all.

The real consequence: preview models will still drop connections intermittently, and the client has no primitive to resume a dropped session. Reconnection has to be implemented at the application level. The parameter was removed from LiveConnectConfig so the session could initialize cleanly on a public API key.

Lesson: session resumption is unavailable on the Developer API — plan for application-level reconnection from the start.

Reconnection Must Happen at the main() Level

With no built-in resumption, the client still dropped periodically. The natural impulse was to retry inside ws_sender_task, but the session object returned by client.aio.live.connect() is bound to the dead WebSocket. It cannot be reused. Retrying inside any of the three worker tasks would either leak a dead session or block the surviving tasks.

The fix was an outer while True loop wrapping the entire async with client.aio.live.connect(...) block, with exponential backoff. Critically, the inner wait uses asyncio.wait(..., return_when=FIRST_EXCEPTION) instead of asyncio.gather(), so the client can tell which task died:

done, pending = await asyncio.wait(
    [mic_task, send_task, output_task],
    return_when=asyncio.FIRST_EXCEPTION
)

for task in pending:
    task.cancel()
await asyncio.gather(*pending, return_exceptions=True)

# Mic death = fatal. WS death = retryable.
for task in done:
    if task is mic_task:
        raise task.exception() or Exception("Mic died")

Backoff was exponential with a 30-second cap, and reset to 1s on each successful connection to avoid reconnect storms. A session resumption handle was stored and passed back on reconnect so the conversation context survived the drop even without Vertex-grade resumption.

Lesson: distinguish fatal hardware failures from retryable transport failures, and own the reconnect loop at the top of the program.

Internal Reasoning Emitted as Text Instead of Audio

On gemini-2.5-flash-native-audio-preview-12-2025, the model intermittently emitted chain-of-thought text in place of audio — strings like "Formulating Polite Reply" or "Seeking Clarification on Issue" — and then produced no audio at all. The transport sat idle, and the same 1011 followed.

This is a model-side bug, not a transport bug. No client-side change resolves it. Switching back to gemini-3.1-flash-live-preview removed the reasoning-text leak, but did not eliminate the periodic connection drops. The transport layer was now correct, and the model was still the remaining variable.

Lesson: when a preview model leaks internal reasoning as text and produces no audio, the correct response is to change models — not to chase the resulting transport symptom.

UNPARSABLE_CLIENT_MESSAGE on the Greeting Field

The first Settings message was rejected with Error parsing client message. Check the greeting field against the API spec. Two mistakes stacked here. First, the greeting field was placed at the top level of the Settings payload rather than inside the agent object. Second, the typed SDK classes in deepgram-sdk 7.10.0 produce a payload the server rejects when the field shape is wrong.

The fix was to place greeting inside AgentV1SettingsAgent and send via the typed send_settings() method, which the SDK expects:

agent=AgentV1SettingsAgent(
    language="en",
    listen=...,
    think=...,
    speak=...,
    greeting="Hello! How can I help you today?",
)

Lesson: the Deepgram server uses deny_unknown_fields. Any field placed at the wrong nesting level produces UNPARSABLE_CLIENT_MESSAGE — read the rejection as a schema error, not a network error.

No Generic send() on V1SocketClient

An attempt to bypass the typed classes by sending raw JSON via connection.send(json.dumps(settings)) failed with AttributeError: 'V1SocketClient' object has no attribute 'send'. Did you mean: '_send'?

The Deepgram Python SDK exposes only typed methods on the socket client — send_settings(), send_media(), send_keep_alive(), and so on. There is no generic send. The fix was to revert to the typed path and correct the field nesting, which resolved the original parse error as well.

Lesson: the Deepgram SDK is typed by design. Work with the provided methods rather than trying to hand-roll the payload.