Passive Interaction Development Guide
Passive Interaction Development Guide
Passive interaction is the core capability of the SDK: the robot initiates a request (audio / video / text) → your SDK callback handles it → you reply via the Response object.
8 passive callback types
Pick the callback type that matches your input/output shape. Each SDK instance may only register one passive callback.
Audio-only
| Callback | Input | Output | Typical scenario |
|---|---|---|---|
Audio2LlmCallback | Audio stream | ASR + LLM text | Voice → text reply |
Audio2TtsCallback | Audio stream | ASR + LLM + TTS | Voice → voice reply |
Text-only
| Callback | Input | Output | Typical scenario |
|---|---|---|---|
Asr2LlmCallback | ASR text | LLM text | Text dialog |
Asr2TtsCallback | ASR text | LLM + TTS | Text → voice reply |
Multimodal (audio+video / text+video)
| Callback | Input | Output | Typical scenario |
|---|---|---|---|
AsrVideo2VlmCallback | ASR text + video | VLM text | Visual QA |
AsrVideo2TtsCallback | ASR text + video | VLM + TTS | Visual QA with voice |
AudioVideo2VlmCallback | Audio + video | ASR + VLM | Full multimodal understanding |
AudioVideo2TtsCallback | Audio + video | ASR + VLM + TTS | Full multimodal end-to-end |
v1.4.0: the
Audio2Asr/Audio2Audio/AudioVideo2Audio/Text2Ttscallbacks and their Response classes have been removed from the SDK.
Flag values
Audio callbacks (Audio2*, AudioVideo2*)
| flag | Meaning | buf | Trigger |
|---|---|---|---|
0 | VAD start | None | User starts speaking |
1 | Audio frame | PCM bytes | Continuous |
2 | VAD end | None | User stops speaking |
3 | H264 video frame | H264 bytes (AudioVideo2* only) | Video stream |
ASR+Video callbacks (AsrVideo2*)
| flag | Meaning | Valid args | Trigger |
|---|---|---|---|
3 | ASR text | text valid, buf=None | ASR done |
4 | H264 video frame | buf valid, text=None | Video stream |
5 | Image frame | buf valid, text=None | Image upload |
The
AsrVideo2*on_requestis invoked multiple times — once for the text and once per video frame; switch onflagto know which one this call carries.
Development template
from linksoul_agentsdk import AgentMeta, AgentParam, AgentSdk
from linksoul_agentsdk.passive import Audio2TtsCallback, Audio2TtsResponse
class MyCallback(Audio2TtsCallback):
def on_robot_online(self, agent_id: str, agent_meta: AgentMeta) -> None:
print(f"online => {agent_id}, wakeup={agent_meta.get_wakeup_word()}")
def on_robot_offline(self, agent_id: str) -> None:
print(f"offline => {agent_id}")
def on_request(
self,
agent_id: str,
event_id: str,
item_id: str,
flag: int,
buf: bytes | None,
param: AgentParam,
response: Audio2TtsResponse,
) -> None:
if flag == 0:
init_asr_session(agent_id, event_id)
elif flag == 1:
feed_asr(agent_id, event_id, buf)
elif flag == 2:
asr_text = finalize_asr(agent_id, event_id)
response.on_asr_final(event_id, asr_text)
response.on_interrupt(event_id, "chat", None)
llm_reply = call_llm(asr_text)
response.on_llm_item_delta(event_id, item_id, llm_reply)
response.on_llm_item_done(event_id, item_id)
response.on_llm_done(event_id)
tts_audio = synthesize_tts(llm_reply)
response.on_tts_item_delta(event_id, item_id, tts_audio)
response.on_tts_item_done(event_id, item_id)
response.on_tts_done(event_id)
sdk = AgentSdk.create(url, app_id, app_key, app_secret)
sdk.register_audio2_tts(MyCallback(sdk))
sdk.initialize()
Response methods at a glance
| Method | Description |
|---|---|
on_asr_middle(event_id, text) | Streaming ASR partial |
on_asr_final(event_id, text) | ASR final |
on_llm_item_delta(event_id, item_id, text) | LLM streaming chunk |
on_llm_item_done(event_id, item_id) | One LLM item done |
on_llm_done(event_id) | All LLM items done |
on_vlm_item_delta(event_id, item_id, text) | VLM streaming chunk |
on_vlm_item_done(event_id, item_id) | One VLM item done |
on_vlm_done(event_id) | All VLM items done |
on_tts_item_delta(event_id, item_id, audio) | TTS audio chunk (bytes) |
on_tts_item_done(event_id, item_id) | One TTS item done |
on_tts_done(event_id) | All TTS items done |
on_skill(event_id, item_id, skill_type, skill_name, param) | Dispatch skill; see Semantic Skills |
on_interrupt(event_id, type, tips) | Interrupt current playback; type ∈ risk_control / action / chat / realtime; tips required only for risk_control. See Interrupts |
on_error(event_id, error_code, error_msg) | Report a failure |
Response capability matrix
| Response | ASR | LLM | VLM | TTS | Skill | Interrupt |
|---|---|---|---|---|---|---|
Audio2LlmResponse | ✓ | ✓ | ✓ | ✓ | ||
Audio2TtsResponse | ✓ | ✓ | ✓ | ✓ | ✓ | |
Asr2LlmResponse | ✓ | ✓ | ✓ | |||
Asr2TtsResponse | ✓ | ✓ | ✓ | ✓ | ||
AsrVideo2VlmResponse | ✓ | ✓ | ✓ | |||
AsrVideo2TtsResponse | ✓ | ✓ | ✓ | ✓ | ||
AudioVideo2VlmResponse | ✓ | ✓ | ✓ | ✓ | ||
AudioVideo2TtsResponse | ✓ | ✓ | ✓ | ✓ | ✓ |
Standard flow
on_request(flag=0) → initialise
on_request(flag=1) → accumulate audio → streaming ASR
→ on_asr_middle() (optional)
on_request(flag=2) → ASR done → on_asr_final()
→ 语义理解 classification
├─ rejected → return (no response.on_*)
└─ valid intent
1. on_interrupt(...) (REQUIRED first)
2. on_skill / on_llm_* / on_vlm_* / on_tts_*
When to dispatch a skill
Trigger: once you have an ASR text, run 语义理解. If the intent maps to a skill
(movement / action / expression / ...), dispatch via response.on_skill(...). Otherwise stay on
the LLM/VLM reply path — do not call on_skill.
Text sources:
| Callback type | Source of text | When to classify |
|---|---|---|
Asr2Llm / Asr2Tts / AsrVideo2Vlm / AsrVideo2Tts | Upstream already did ASR; you get text directly | As soon as text arrives |
Audio2Llm / Audio2Tts / AudioVideo2Vlm / AudioVideo2Tts | SDK accumulates flag=1 audio and finalises at flag=2 | After computing the final text in flag=2 |
Typical flow (Asr2Llm):
on_request(text="walk forward 10 meters")
│
▼
语义理解 classification
├─ rejected → return — no response.on_* calls
│
└─ valid intent
1. response.on_interrupt(event_id, type, tips)
(type ∈ risk_control / action / chat / realtime)
2. fill in skill / LLM / VLM / TTS
⚠️ Strict ordering: once classified as valid, you MUST call
on_interruptfirst, then any ofon_skill/on_llm_*/on_vlm_*/on_tts_*. Reversing the order leaves the robot stuck on the previous reply. The only exception is rejected — call nothing.
Example:
def on_request(self, agent_id, event_id, text, param, response):
nlu = classify(text)
# 1) rejected → return
if nlu.is_rejected():
return
item_id = IdGenerator.generate_item_id()
# 2) MUST interrupt first
if nlu.is_risk_control():
response.on_interrupt(
event_id, "risk_control",
"You mentioned a sensitive topic; let's talk about something else",
)
return # typically no further LLM / skill output after risk control
elif nlu.is_skill():
response.on_interrupt(event_id, "action", None)
else:
response.on_interrupt(event_id, "chat", None)
# 3) Skill (optional)
if nlu.is_skill():
skill_param = (
AgentParam.create()
.set_string("movement", "move_forward")
.set_integer("meter", 10)
)
response.on_skill(event_id, item_id, "movement", "move_forward", skill_param)
# 4) LLM reply
response.on_llm_item_delta(event_id, item_id, "Sure, walking forward.")
response.on_llm_item_done(event_id, item_id)
response.on_llm_done(event_id)
Interrupts (on_interrupt)
response.on_interrupt(event_id, interrupt_type, interrupt_tips) is called before a new reply
so that the robot stops whatever it is currently doing (playback / motion / conversation) before
the new TTS / skill / dialog content starts.
interrupt_type values
| Value | Meaning | Scenario | interrupt_tips |
|---|---|---|---|
risk_control | Risk-control interrupt | Sensitive content; must cut current dialog | Required, e.g. "You mentioned a sensitive topic; let's talk about something else" |
action | Skill interrupt | 语义理解 matched a skill / action — barge in before dispatching the new action | Pass None |
chat | Valid-chat interrupt | New LLM/VLM text reply cuts off the old one | Pass None |
realtime | Realtime interrupt | Talk-while-listening; new segment cuts old | Pass None |
Call order (mandatory)
⚠️ For every valid intent, the response sequence is mandatory:
response.on_interrupt(event_id, type, tips)response.on_skill(...)and / orresponse.on_llm_item_* / on_vlm_item_* / on_tts_item_*
| 语义理解 result | First call | Allowed subsequent replies |
|---|---|---|
| Risk-control | on_interrupt(..., "risk_control", "<spoken prompt>") | Typically nothing further (business-dependent) |
| Skill | on_interrupt(..., "action", None) | on_skill(...) (can run in parallel with LLM/VLM/TTS) |
| Valid chat | on_interrupt(..., "chat", None) | on_llm_item_* / on_vlm_item_* / on_tts_item_* |
| Realtime | on_interrupt(..., "realtime", None) | Realtime reply |
| Rejected | Do not call | No response.on_* at all |
Examples
# Plain chat reply
response.on_interrupt(event_id, "chat", None)
response.on_llm_item_delta(event_id, item_id, "Sure, here is the answer.")
# Action / skill
response.on_interrupt(event_id, "action", None)
skill_param = AgentParam.create().set_string("action", "wave_hands")
response.on_skill(event_id, item_id, "action", "wave_hands", skill_param)
# Realtime dialog
response.on_interrupt(event_id, "realtime", None)
response.on_llm_item_delta(event_id, item_id, "Mhm, I hear you...")
# Risk-control (tips is mandatory — used as the spoken reply)
response.on_interrupt(event_id, "risk_control",
"You mentioned a sensitive topic; let's talk about something else")
Shared base callbacks (v1.4.0+)
PassiveCallback declares mode-agnostic hooks inherited by every concrete subclass; override
whichever you need (otherwise the default log-only implementation runs):
| Method | Trigger | Notes |
|---|---|---|
on_robot_online(agent_id, agent_meta) | Agent online | agent_id is the agent ID configured on the LinkSoul platform; agent_meta carries device metadata (wake word, city, etc.) |
on_robot_offline(agent_id) | Agent offline | Release resources associated with this agent_id |
on_face_info(agent_id, event_id, param) | Face / voiceprint UID detected | param carries the fields |
on_video_frame(agent_id, event_id, flag, buf, is_key_frame, local_ts, param) | Video passthrough | flag=2 H264, flag=3 image |
on_greet_signal(agent_id, event_id, param, response) | Greeting signal | Reply via GreetResponse |
on_state(agent_id, event_id, state_name, state_value) | Robot state push | state_value may be JSON; see robot-side state modules for every state_name |
Greeting reply (GreetResponse)
def on_greet_signal(self, agent_id, event_id, param, response):
# Mode 1: streamed VLM text
response.on_greet_vlm_delta(event_id, "Hello")
response.on_greet_vlm_delta(event_id, ", nice to see you")
response.on_greet_vlm_done(event_id)
# Mode 2: streamed TTS audio — argument is a base64-encoded chunk of synthesised audio bytes
import base64
audio_chunk1 = synthesize_tts("Hello") # PCM / Opus / ... bytes
audio_chunk2 = synthesize_tts(", welcome home")
response.on_greet_tts_delta(event_id, base64.b64encode(audio_chunk1).decode())
response.on_greet_tts_delta(event_id, base64.b64encode(audio_chunk2).decode())
response.on_greet_tts_done(event_id)
| Method | Description |
|---|---|
on_greet_vlm_delta(event_id, text) | VLM streaming chunk |
on_greet_vlm_done(event_id) | VLM finished |
on_greet_tts_delta(event_id, audio_base64) | TTS audio chunk — audio_base64 is the base64-encoded chunk of synthesised audio bytes |
on_greet_tts_done(event_id) | TTS audio stream finished |
on_error(event_id, code, msg) | Error reply |
Important notes
- Callback is a singleton: every online agent shares one instance; use
agent_id(the agent ID configured on the LinkSoul platform) to distinguish. - Threading: each
agent_idbinds to a dedicated worker; requests from one agent run in order. event_idisolation: manage non-thread-safe per-turn state keyed byevent_id.- Response is thread-safe:
response.on_*()may be called from any thread. - Response TTL cleanup:
AgentSdkResponseMgrauto-evicts responses 120 s after the last touch. - Register before initialize:
register_*()must be called beforeinitialize(). - Override only what you need: every base callback has a default log-only implementation.