Agentic VoiceDSP module

Agentic VoiceDSP combines Silero VAD (voice activity detection), Pipecat turn detection, and Hush noise suppression in a single VoxEngine module, ready to use in your voice application.

While you can import all these modules one-by-one in your scenario, Agentic VoiceDSP provides an all-in-one solution.

This module is a perfect ready-to-use solution if you want to create your own LLM-application, however it is useful with any speech processing applications.

How it works
Usage

How it works

This module provides a client interface to communicate with a bundled voice DSP connector in real time (over WebSocket). The connector combines Silero VAD voice-activity detection, Pipecat Turn turn detection, and optional Hush noise suppression, and applies configurable start/stop strategies to produce high-level user-turn signals for agentic applications.

Connector component	Vendor / stack	Scenario-facing config
VAD	Silero	`vadParameters` ([Agentic.VADParameters])
Turn detector	Pipecat Smart Turn	`turnDetectorParameters` ([Agentic.TurnDetectorParameters])
Noise suppression	Hush (ONNX, PCM16 @ 16 kHz)	`noiseSuppressionParameters` ([Agentic.NoiseSuppressionParameters]) — optional; off unless provided

Turn lifecycle mirrors Pipecat UserTurnProcessor and TurnAnalyzerUserTurnStopStrategy.

Smart Turn flow

VAD detects a pause → the turn detector runs (Predict).
If the model returns end of turn → TurnInferenceTriggered, then TurnStopped (after transcript gating when ASR is configured).
If the model returns incomplete, the silence should continue for stopSecs seconds → TurnStopTimeout (reason: stop_secs), then TurnStopped.

If the user resumes speaking before the Predict response arrives, that response is ignored (in-flight Predict is invalidated on VAD speech start). This prevents a stale endOfTurn from emitting TurnInferenceTriggered / merging the next utterance into the same turn.

stopSecs is not forwarded to the connector; it is handled in voice-dsp-impl.ts.

Stop-strategy precedence

When multiple stop strategies are configured, they are ranked so a single turn always finalizes with one strategy and TurnInferenceTriggered.strategy === TurnStopped.strategy:

controller_timeout (the user-turn-stop watchdog force-stop) — always has priority; it finalizes a stuck turn even when a transcript is still awaited, reusing the pending strategy label (or the configured fallback if nothing is pending).
turn_detector (the ML verdict, plus its stop_secs fallback that is emitted with strategy: 'turn_detector') — authoritative.
speech_timeout — a pure fallback.

A speech_timeout fallback never pre-empties or relabels the ML turn_detector while it is engaged for the current segment (a Predict verdict is pending, a turn_detector finalize is already pending, or the stop_secs fallback is armed after an endOfTurn: false verdict). Consequently, when turn_detector and speech_timeout are configured together, the detector (with its stop_secs fallback and the controller_timeout watchdog) owns every finalization path, so speech_timeout only fires when turn_detector is not configured.

TurnInferenceTriggered.strategy and TurnStopped.strategy always match a configured stop strategy. If turn_detector is omitted from stopStrategies (tests, or ASR-only stop logic):

VoiceDSP does not send Predict.
Connector Turn.Result is ignored.
Events never report strategy: 'turn_detector'. The watchdog still force-stops with TurnStopTimeout reason: controller_timeout, labeled with the configured fallback (speech_timeout).

ASR transcript gating

When an ASR instance is passed to Agentic.createVoiceDSP:

Transcription-based start strategies (transcription, min_words) can emit TurnStarted.
TurnStopped waits for a final transcript when possible.
If the turn analyzer completes before ASR, TurnInferenceTriggered fires first so the scenario can start LLM inference early.
An ASR grace timer (asrGraceTimeout, default 0.3) starts at TurnInferenceTriggered. Until it expires, later ASR interim or final results update the snapshot used by TurnStopped. A final transcript finalizes the turn immediately. Keep the default shorter than typical ASR final latency so TurnStopped does not wait for ASR final; start LLM on TurnInferenceTriggered. When the timer expires, TurnStopped uses the latest snapshot (including interim text).
After TurnStopped, late ASR interim/final for the closed utterance is ignored until a new speech segment (VAD speech start) or a new turn start (prevents ghost TurnStarted / content bleed into the next turn).

** Pass the same ASR instance you send call media to: Agentic.createVoiceDSP({ asr, startStrategies: [Agentic.VoiceDSPStrategies.createMinWordsStart({ minWords: 3 })] }). Without asr, those start strategies never fire. Do not rely on instanceof across Modules.ASR / Modules.Agentic — VoiceDSP accepts any ASR-like object with event listeners.

User turn stop watchdog

userTurnStopTimeout (default 5.0 s, mirrors Pipecat user_turn_stop_timeout) forces turn finalization when a turn stays active without a stop strategy firing. Emits TurnStopTimeout with reason: controller_timeout, then TurnStopped. The inference and stop events reuse the pending strategy, or the configured fallback if none is pending — never turn_detector unless that strategy is configured.

User idle

Mirrors Pipecat UserIdleController (separate from Smart Turn stopSecs):

Set userTurnIdleTimeout (seconds; 0 = disabled).
Call signalAssistantTurnEnded() when TTS/LLM playback finishes → clears assistant-speaking state and starts the idle timer if the user is not in an active turn.
Call signalAssistantTurnStarted() when the assistant starts speaking → marks assistant-speaking and cancels the idle timer.
TurnStarted.interrupted is true only when the user turn starts while the assistant is speaking (requires the signal calls above). It is not a copy of strategy enableInterruptions.
When the assistant is speaking and a start strategy has enableInterruptions: false, TurnStarted is suppressed.
User speech or TurnStarted also cancel the idle timer.

Noise suppression (Hush)

Pass noiseSuppressionParameters to [Agentic.createVoiceDSP] to enable Hush noise suppression inside the bundled /agentic/voice_dsp connector (same ONNX stack as standalone Hush.NoiseSuppression).

Opt-in only: omit noiseSuppressionParameters to leave noise suppression disabled.
Parameter shape matches [Hush.NoiseSuppressionParameters] (attenLimDb; developer-only model / cpuCount stay @hidden).
Do not require(Modules.Hush) for this path — Hush runs server-side in the VoiceDSP connector, not as a separate Hush.createNoiseSuppression media unit. Standalone Hush remains available when you need an explicit duplex NS unit.
Denoised audio stays inside the connector pipeline before VAD / turn detection; scenarios still send call media only to Agentic.VoiceDSP.

Usage

Create an Agentic.VoiceDSP instance (via the Agentic.createVoiceDSP method) and provide your parameters
Send media between Call and Agentic.VoiceDSP via the call.sendMediaTo method
Optionally pass an ASR instance when using transcription-based start strategies (transcription, min_words)
Set turnDetectorParameters.stopSecs (default 3.0)
Optionally set asrGraceTimeout, userTurnStopTimeout, userTurnIdleTimeout
Wire signalAssistantTurnStarted / signalAssistantTurnEnded to your TTS/LLM lifecycle
Configure start strategies and stop strategies via [Agentic.VoiceDSPStrategies] factories
Listen to Agentic.VoiceDSPEvents and implement your application business logic

Here is the complete VoxEngine scenario showing how to use Agentic VoiceDSP module for your reference: