Passive Interaction Development Guide

Passive Interaction Development Guide

← Home

Passive interaction is the core capability of the SDK: the robot initiates a request (audio / video / text) → your SDK callback handles it → you reply via the Response object.

8 passive callback types

Pick the callback type that matches your input/output shape. Each SDK instance may only register one passive callback.

Audio-only

CallbackInputOutputTypical scenario
Audio2LlmCallbackAudio streamASR + LLM textVoice → text reply
Audio2TtsCallbackAudio streamASR + LLM + TTSVoice → voice reply

Text-only

CallbackInputOutputTypical scenario
Asr2LlmCallbackASR textLLM textText dialog
Asr2TtsCallbackASR textLLM + TTSText → voice reply

Multimodal (audio+video / text+video)

CallbackInputOutputTypical scenario
AsrVideo2VlmCallbackASR text + videoVLM textVisual QA
AsrVideo2TtsCallbackASR text + videoVLM + TTSVisual QA with voice
AudioVideo2VlmCallbackAudio + videoASR + VLMFull multimodal understanding
AudioVideo2TtsCallbackAudio + videoASR + VLM + TTSFull multimodal end-to-end

v1.4.0: the Audio2Asr / Audio2Audio / AudioVideo2Audio / Text2Tts callbacks and their Response classes have been removed from the SDK.

Flag values

Audio callbacks (Audio2*, AudioVideo2*)

flagMeaningbufTrigger
0VAD startNoneUser starts speaking
1Audio framePCM bytesContinuous
2VAD endNoneUser stops speaking
3H264 video frameH264 bytes (AudioVideo2* only)Video stream

ASR+Video callbacks (AsrVideo2*)

flagMeaningValid argsTrigger
3ASR texttext valid, buf=NoneASR done
4H264 video framebuf valid, text=NoneVideo stream
5Image framebuf valid, text=NoneImage upload

The AsrVideo2* on_request is invoked multiple times — once for the text and once per video frame; switch on flag to know which one this call carries.

Development template

python
from linksoul_agentsdk import AgentMeta, AgentParam, AgentSdk
from linksoul_agentsdk.passive import Audio2TtsCallback, Audio2TtsResponse


class MyCallback(Audio2TtsCallback):

    def on_robot_online(self, agent_id: str, agent_meta: AgentMeta) -> None:
        print(f"online => {agent_id}, wakeup={agent_meta.get_wakeup_word()}")

    def on_robot_offline(self, agent_id: str) -> None:
        print(f"offline => {agent_id}")

    def on_request(
        self,
        agent_id: str,
        event_id: str,
        item_id: str,
        flag: int,
        buf: bytes | None,
        param: AgentParam,
        response: Audio2TtsResponse,
    ) -> None:
        if flag == 0:
            init_asr_session(agent_id, event_id)
        elif flag == 1:
            feed_asr(agent_id, event_id, buf)
        elif flag == 2:
            asr_text = finalize_asr(agent_id, event_id)
            response.on_asr_final(event_id, asr_text)
            response.on_interrupt(event_id, "chat", None)
            llm_reply = call_llm(asr_text)
            response.on_llm_item_delta(event_id, item_id, llm_reply)
            response.on_llm_item_done(event_id, item_id)
            response.on_llm_done(event_id)
            tts_audio = synthesize_tts(llm_reply)
            response.on_tts_item_delta(event_id, item_id, tts_audio)
            response.on_tts_item_done(event_id, item_id)
            response.on_tts_done(event_id)


sdk = AgentSdk.create(url, app_id, app_key, app_secret)
sdk.register_audio2_tts(MyCallback(sdk))
sdk.initialize()

Response methods at a glance

MethodDescription
on_asr_middle(event_id, text)Streaming ASR partial
on_asr_final(event_id, text)ASR final
on_llm_item_delta(event_id, item_id, text)LLM streaming chunk
on_llm_item_done(event_id, item_id)One LLM item done
on_llm_done(event_id)All LLM items done
on_vlm_item_delta(event_id, item_id, text)VLM streaming chunk
on_vlm_item_done(event_id, item_id)One VLM item done
on_vlm_done(event_id)All VLM items done
on_tts_item_delta(event_id, item_id, audio)TTS audio chunk (bytes)
on_tts_item_done(event_id, item_id)One TTS item done
on_tts_done(event_id)All TTS items done
on_skill(event_id, item_id, skill_type, skill_name, param)Dispatch skill; see Semantic Skills
on_interrupt(event_id, type, tips)Interrupt current playback; typerisk_control / action / chat / realtime; tips required only for risk_control. See Interrupts
on_error(event_id, error_code, error_msg)Report a failure

Response capability matrix

ResponseASRLLMVLMTTSSkillInterrupt
Audio2LlmResponse
Audio2TtsResponse
Asr2LlmResponse
Asr2TtsResponse
AsrVideo2VlmResponse
AsrVideo2TtsResponse
AudioVideo2VlmResponse
AudioVideo2TtsResponse

Standard flow

text
on_request(flag=0) → initialise
on_request(flag=1) → accumulate audio → streaming ASR
                                       → on_asr_middle()  (optional)
on_request(flag=2) → ASR done → on_asr_final()
                              → 语义理解 classification
                                ├─ rejected → return (no response.on_*)
                                └─ valid intent
                                     1. on_interrupt(...)  (REQUIRED first)
                                     2. on_skill / on_llm_* / on_vlm_* / on_tts_*

When to dispatch a skill

Trigger: once you have an ASR text, run 语义理解. If the intent maps to a skill (movement / action / expression / ...), dispatch via response.on_skill(...). Otherwise stay on the LLM/VLM reply path — do not call on_skill.

Text sources:

Callback typeSource of textWhen to classify
Asr2Llm / Asr2Tts / AsrVideo2Vlm / AsrVideo2TtsUpstream already did ASR; you get text directlyAs soon as text arrives
Audio2Llm / Audio2Tts / AudioVideo2Vlm / AudioVideo2TtsSDK accumulates flag=1 audio and finalises at flag=2After computing the final text in flag=2

Typical flow (Asr2Llm):

text
on_request(text="walk forward 10 meters")
      │
      ▼
语义理解 classification
      ├─ rejected → return — no response.on_* calls
      │
      └─ valid intent
           1. response.on_interrupt(event_id, type, tips)
              (type ∈ risk_control / action / chat / realtime)
           2. fill in skill / LLM / VLM / TTS

⚠️ Strict ordering: once classified as valid, you MUST call on_interrupt first, then any of on_skill / on_llm_* / on_vlm_* / on_tts_*. Reversing the order leaves the robot stuck on the previous reply. The only exception is rejected — call nothing.

Example:

python
def on_request(self, agent_id, event_id, text, param, response):
    nlu = classify(text)

    # 1) rejected → return
    if nlu.is_rejected():
        return

    item_id = IdGenerator.generate_item_id()

    # 2) MUST interrupt first
    if nlu.is_risk_control():
        response.on_interrupt(
            event_id, "risk_control",
            "You mentioned a sensitive topic; let's talk about something else",
        )
        return  # typically no further LLM / skill output after risk control
    elif nlu.is_skill():
        response.on_interrupt(event_id, "action", None)
    else:
        response.on_interrupt(event_id, "chat", None)

    # 3) Skill (optional)
    if nlu.is_skill():
        skill_param = (
            AgentParam.create()
            .set_string("movement", "move_forward")
            .set_integer("meter", 10)
        )
        response.on_skill(event_id, item_id, "movement", "move_forward", skill_param)

    # 4) LLM reply
    response.on_llm_item_delta(event_id, item_id, "Sure, walking forward.")
    response.on_llm_item_done(event_id, item_id)
    response.on_llm_done(event_id)

Interrupts (on_interrupt)

response.on_interrupt(event_id, interrupt_type, interrupt_tips) is called before a new reply so that the robot stops whatever it is currently doing (playback / motion / conversation) before the new TTS / skill / dialog content starts.

interrupt_type values

ValueMeaningScenariointerrupt_tips
risk_controlRisk-control interruptSensitive content; must cut current dialogRequired, e.g. "You mentioned a sensitive topic; let's talk about something else"
actionSkill interrupt语义理解 matched a skill / action — barge in before dispatching the new actionPass None
chatValid-chat interruptNew LLM/VLM text reply cuts off the old onePass None
realtimeRealtime interruptTalk-while-listening; new segment cuts oldPass None

Call order (mandatory)

⚠️ For every valid intent, the response sequence is mandatory:

  1. response.on_interrupt(event_id, type, tips)
  2. response.on_skill(...) and / or response.on_llm_item_* / on_vlm_item_* / on_tts_item_*
语义理解 resultFirst callAllowed subsequent replies
Risk-controlon_interrupt(..., "risk_control", "<spoken prompt>")Typically nothing further (business-dependent)
Skillon_interrupt(..., "action", None)on_skill(...) (can run in parallel with LLM/VLM/TTS)
Valid chaton_interrupt(..., "chat", None)on_llm_item_* / on_vlm_item_* / on_tts_item_*
Realtimeon_interrupt(..., "realtime", None)Realtime reply
RejectedDo not callNo response.on_* at all

Examples

python
# Plain chat reply
response.on_interrupt(event_id, "chat", None)
response.on_llm_item_delta(event_id, item_id, "Sure, here is the answer.")

# Action / skill
response.on_interrupt(event_id, "action", None)
skill_param = AgentParam.create().set_string("action", "wave_hands")
response.on_skill(event_id, item_id, "action", "wave_hands", skill_param)

# Realtime dialog
response.on_interrupt(event_id, "realtime", None)
response.on_llm_item_delta(event_id, item_id, "Mhm, I hear you...")

# Risk-control (tips is mandatory — used as the spoken reply)
response.on_interrupt(event_id, "risk_control",
    "You mentioned a sensitive topic; let's talk about something else")

Shared base callbacks (v1.4.0+)

PassiveCallback declares mode-agnostic hooks inherited by every concrete subclass; override whichever you need (otherwise the default log-only implementation runs):

MethodTriggerNotes
on_robot_online(agent_id, agent_meta)Agent onlineagent_id is the agent ID configured on the LinkSoul platform; agent_meta carries device metadata (wake word, city, etc.)
on_robot_offline(agent_id)Agent offlineRelease resources associated with this agent_id
on_face_info(agent_id, event_id, param)Face / voiceprint UID detectedparam carries the fields
on_video_frame(agent_id, event_id, flag, buf, is_key_frame, local_ts, param)Video passthroughflag=2 H264, flag=3 image
on_greet_signal(agent_id, event_id, param, response)Greeting signalReply via GreetResponse
on_state(agent_id, event_id, state_name, state_value)Robot state pushstate_value may be JSON; see robot-side state modules for every state_name

Greeting reply (GreetResponse)

python
def on_greet_signal(self, agent_id, event_id, param, response):
    # Mode 1: streamed VLM text
    response.on_greet_vlm_delta(event_id, "Hello")
    response.on_greet_vlm_delta(event_id, ", nice to see you")
    response.on_greet_vlm_done(event_id)

    # Mode 2: streamed TTS audio — argument is a base64-encoded chunk of synthesised audio bytes
    import base64
    audio_chunk1 = synthesize_tts("Hello")               # PCM / Opus / ... bytes
    audio_chunk2 = synthesize_tts(", welcome home")
    response.on_greet_tts_delta(event_id, base64.b64encode(audio_chunk1).decode())
    response.on_greet_tts_delta(event_id, base64.b64encode(audio_chunk2).decode())
    response.on_greet_tts_done(event_id)
MethodDescription
on_greet_vlm_delta(event_id, text)VLM streaming chunk
on_greet_vlm_done(event_id)VLM finished
on_greet_tts_delta(event_id, audio_base64)TTS audio chunk — audio_base64 is the base64-encoded chunk of synthesised audio bytes
on_greet_tts_done(event_id)TTS audio stream finished
on_error(event_id, code, msg)Error reply

Important notes

  1. Callback is a singleton: every online agent shares one instance; use agent_id (the agent ID configured on the LinkSoul platform) to distinguish.
  2. Threading: each agent_id binds to a dedicated worker; requests from one agent run in order.
  3. event_id isolation: manage non-thread-safe per-turn state keyed by event_id.
  4. Response is thread-safe: response.on_*() may be called from any thread.
  5. Response TTL cleanup: AgentSdkResponseMgr auto-evicts responses 120 s after the last touch.
  6. Register before initialize: register_*() must be called before initialize().
  7. Override only what you need: every base callback has a default log-only implementation.