Passive Interaction Development Guide

Passive Interaction Development Guide

Passive interaction is the core functionality of the SDK: Robot initiates a request (audio/video/text) -> SDK callback processes it -> Returns result via Response.

8 Passive Callback Types

Choose the appropriate callback type based on input/output combination. Each SDK instance can register only one passive callback.

Audio-Only

Callback ClassInputOutputTypical Scenario
Audio2LlmCallbackAudio streamASR + LLM textVoice conversation, text reply
Audio2TtsCallbackAudio streamASR + LLM + TTS audioVoice conversation, voice reply

Text-Only

Callback ClassInputOutputTypical Scenario
Asr2LlmCallbackASR textLLM textText conversation
Asr2TtsCallbackASR textLLM + TTS audioText to voice reply

Multimodal (Audio+Video / Text+Video)

Callback ClassInputOutputTypical Scenario
AsrVideo2VlmCallbackASR text + VideoVLM textImage understanding
AsrVideo2TtsCallbackASR text + VideoVLM + TTS audioImage understanding with voice reply
AudioVideo2VlmCallbackAudio + VideoASR + VLM textFull multimodal understanding
AudioVideo2TtsCallbackAudio + VideoASR + VLM + TTSFull multimodal voice reply

v1.4.0: the Audio2Asr / Audio2Audio / AudioVideo2Audio / Text2Tts callbacks and their Response classes have been removed from the SDK.

Flag Values Explained

flag is the key parameter for distinguishing different data types within the same callback.

Audio Callbacks (Audio2*, AudioVideo2*)

flagMeaningbuf ContentTrigger Condition
0VAD startnullUser starts speaking
1Audio framePCM audio dataContinuous transmission
2VAD endnullUser stops speaking
4H264 video frameH264 encoded dataVideo stream transmission
5Image frameJPEG/PNG imageImage transmission

ASR+Video Callbacks (AsrVideo2*)

flagMeaningValid ParametersTrigger Condition
3ASR texttext is valid, buf=nullASR recognition complete
4H264 video framebuf is valid, text=nullVideo stream transmission
5Image framebuf is valid, text=nullImage transmission

Key Understanding: The onRequest method of AsrVideo2* callbacks will be called multiple times (separately delivering text and video). You need to check the flag value to determine what data is being passed in each call.

Development Template

java
agentSdk.registerAudio2Tts(new Audio2TtsCallback(agentSdk) {

    @Override
    public void onRobotOnline(String agentId, AgentMeta agentMeta) {
        // Robot comes online, initialize resources for this robot
        // agentMeta contains wakeup word, city, and other configuration
        String wakeupWord = agentMeta.getWakeupWord();
    }

    @Override
    public void onRobotOffline(String agentId) {
        // Robot goes offline, clean up resources for this robot
    }

    @Override
    public void onRequest(String agentId, String eventId, String itemId,
                          int flag, byte[] audio, AgentParam param,
                          Audio2TtsResponse response) {
        switch (flag) {
            case 0 -> {
                // VAD start: initialize ASR session
            }
            case 1 -> {
                // Audio frame: send to ASR service for streaming recognition
                sendToAsr(audio);
            }
            case 2 -> {
                // VAD end: finalize ASR, call LLM, synthesize TTS
                String asrText = finalizeAsr();
                response.onAsrFinal(eventId, asrText);

                // Call LLM to get reply
                String llmReply = callLlm(asrText);
                response.onLlmItemDelta(eventId, itemId, llmReply);
                response.onLlmItemDone(eventId, itemId);
                response.onLlmDone(eventId);

                // Synthesize TTS audio
                byte[] ttsAudio = synthesizeTts(llmReply);
                response.onTtsItemDelta(eventId, itemId, ttsAudio);
                response.onTtsItemDone(eventId, itemId);
                response.onTtsDone(eventId);
            }
        }
    }
});

Response Methods Overview

All Response classes inherit from AgentSdkResponse. Available method combinations differ by callback type:

MethodDescriptionParameters
onAsrMiddle(eventId, text)ASR intermediate result (streaming)text: currently recognized text
onAsrFinal(eventId, text)ASR final resulttext: complete recognized text
onLlmItemDelta(eventId, itemId, text)LLM streaming outputtext: incremental text for this chunk
onLlmItemDone(eventId, itemId)LLM single segment complete
onLlmDone(eventId)LLM all segments complete
onVlmItemDelta(eventId, itemId, text)VLM streaming outputtext: incremental text for this chunk
onVlmItemDone(eventId, itemId)VLM single segment complete
onVlmDone(eventId)VLM all segments complete
onTtsItemDelta(eventId, itemId, audio)TTS audio chunkaudio: PCM byte array
onTtsItemDone(eventId, itemId)TTS single segment complete
onTtsDone(eventId)TTS all segments complete
onSkill(eventId, itemId, skillType, skillName, param)Dispatch skill commandparam: AgentParam; see Semantic Skills for the full catalogue
onInterrupt(eventId, type, tips)Interrupt current conversationtyperisk_control / action / chat / realtime; tips required only for risk_control. See Interrupts
onError(eventId, errorCode, errorMsg)Return errorSupported by all Response types

Response Available Methods Matrix

Response ClassASRLLMVLMTTSSkillInterrupt
Audio2LlmResponse
Audio2TtsResponse
Asr2LlmResponse
Asr2TtsResponse
AsrVideo2VlmResponse
AsrVideo2TtsResponse
AudioVideo2VlmResponse
AudioVideo2TtsResponse

Standard Response Flow

text
onRequest(flag=0) -> Initialize
onRequest(flag=1) -> Accumulate audio -> ASR streaming recognition
                                       -> onAsrMiddle() (optional, intermediate result)
onRequest(flag=2) -> ASR complete -> onAsrFinal()
                                  -> LLM call -> onLlmItemDelta() x N
                                              -> onLlmItemDone()
                                              -> onLlmDone()
                                  -> TTS synthesis -> onTtsItemDelta() x N
                                                   -> onTtsItemDone()
                                                   -> onTtsDone()

When to dispatch a skill

Trigger: once you have an ASR text, run 语义理解 (intent classification) on it. If the intent maps to a skill (movement, action, expression, ...), dispatch it through response.onSkill(...). If it does not, just continue with the normal LLM/VLM text reply — do not call onSkill.

There are two ways the ASR text arrives:

Callback typeWhere the text comes fromWhen to classify
Asr2Llm / Asr2Tts / AsrVideo2Vlm / AsrVideo2TtsUpstream (the robot side) already finished ASR and hands you the final text directly through onRequest(... text ...)As soon as text arrives
Audio2Llm / Audio2Tts / AudioVideo2Vlm / AudioVideo2TtsThe SDK side accumulates flag=1 audio frames and produces the final text at flag=2 (VAD end)After computing the final text — typically right after response.onAsrFinal in the flag=2 branch

Typical flow (Asr2Llm shown):

text
onRequest(text="please walk forward 10 meters")
      │
      ▼
语义理解 classification
      ├─ rejected (invalid query / false wake / noise / ...)
      │    └→ just return — do NOT call any response.on* method
      │       (no skill, no interrupt, no LLM/VLM output)
      │
      └─ valid intent
           ① first MUST call response.onInterrupt(eventId, type, tips)
           │    (type ∈ risk_control / action / chat / realtime — see "Interrupts" below)
           │
           ② then fill in the result:
           ├─ skill hit → response.onSkill(eventId, itemId, ...)
           ├─ valid chat → response.onLlmItemDelta / onLlmItemDone / onLlmDone
           ├─ multimodal → response.onVlmItemDelta / onVlmItemDone / onVlmDone
           └─ TTS reply (when applicable) → response.onTtsItemDelta / onTtsItemDone / onTtsDone

⚠️ Strict ordering: once the intent is classified as valid, you MUST call onInterrupt first, then any of onSkill / onLlm* / onVlm* / onTts*. Reversing the order leaves the robot stuck on the previous reply. The only exception is rejected — call nothing, let the robot keep its current state.

Example (Asr2Llm; the Audio2Tts case does the same 语义理解 step inside its flag=2 branch):

java
@Override
public void onRequest(String agentId, String eventId, String text,
                      AgentParam param, Asr2LlmResponse response) {

    // 语义理解 on the business side: classify `text` into
    // "skill / chat / realtime / risk_control / rejected"
    NluResult nlu = classify(text);

    // ① Rejected: just return — do not call any response.on* method
    if (nlu.isRejected()) {
        return;
    }

    String itemId = IdGenerator.generateItemId();

    // ② Valid intent — MUST interrupt first, then fill in skill / LLM / VLM / TTS
    if (nlu.isRiskControl()) {
        // Risk control: interruptTips is required
        response.onInterrupt(eventId, "risk_control",
                "You mentioned a sensitive topic; let's talk about something else");
        return; // typically no further skill / LLM output after risk control
    } else if (nlu.isSkill()) {
        response.onInterrupt(eventId, "action", null);    // skill interrupt
    } else {
        response.onInterrupt(eventId, "chat", null);      // valid-chat interrupt
        // For realtime dialog: response.onInterrupt(eventId, "realtime", null);
    }

    // ③ Skill hit: dispatch skillParam (keys/values come from the Semantic Skills reference)
    if (nlu.isSkill()) {
        AgentParam skillParam = AgentParam.create()
            .setString("movement", "move_forward")
            .setInteger("meter", 10);
        response.onSkill(eventId, itemId, "movement", "move_forward", skillParam);
    }

    // ④ LLM streaming reply (runs alongside the skill — they do not block each other)
    response.onLlmItemDelta(eventId, itemId, "Sure, walking forward.");
    response.onLlmItemDone(eventId, itemId);
    response.onLlmDone(eventId);
}

skillType / skillName and the contents of skillParam are listed in Semantic Skills. Multiple onSkill calls under the same eventId are allowed (multi-intent scenarios) — use a distinct itemId per intent.

Interrupts (onInterrupt)

response.onInterrupt(eventId, interruptType, interruptTips) is called before a new reply so that the robot stops whatever it is currently doing (playback / motion / conversation) before the new TTS / skill / dialog content starts.

interruptType values

ValueMeaningScenariointerruptTips
risk_controlRisk-control interruptContent moderation, sensitive topic; the current dialog must be cut immediatelyRequired, e.g. "You mentioned a sensitive topic; let's talk about something else"
actionSkill interrupt语义理解 matched a skill / action — barge in on the current motion or playback before dispatching the new actionPass null
chatValid-chat interrupt语义理解 matched plain chat (valid conversation) — a new LLM/VLM text reply starts and cuts off the old onePass null
realtimeRealtime interruptTalk-while-listening (realtime) dialog — a new realtime segment cuts off the old onePass null

interruptTips is a user-facing message. Only risk_control requires it — the robot will speak that line to the user when the risk-control interrupt fires. For the other three types, pass null by convention.

Call order (mandatory)

⚠️ For every valid intent, the response sequence below is mandatory, otherwise the robot cannot enter the new reply state:

  1. response.onInterrupt(eventId, type, tips)
  2. response.onSkill(...) and / or response.onLlmItem* / onVlmItem* / onTtsItem*

Pick the interruptType based on the 语义理解 classification:

语义理解 resultFirst callAllowed subsequent replies
Risk-control hitonInterrupt(..., "risk_control", "<spoken prompt>")Typically no further LLM/VLM/skill replies (business-dependent)
Skill hitonInterrupt(..., "action", null)onSkill(...) (can run in parallel with onLlm* / onVlm* / onTts*)
Valid chatonInterrupt(..., "chat", null)onLlmItem* / onVlmItem* / onTtsItem*
Realtime dialogonInterrupt(..., "realtime", null)Realtime reply content
RejectedDo not call onInterruptCall nothing — let the robot keep its current state

Examples

java
// Plain chat reply
response.onInterrupt(eventId, "chat", null);
response.onLlmItemDelta(eventId, itemId, "Sure, here is the answer.");

// Action / skill hit
response.onInterrupt(eventId, "action", null);
AgentParam skillParam = AgentParam.create().setString("action", "wave_hands");
response.onSkill(eventId, itemId, "action", "wave_hands", skillParam);

// Realtime talk-while-listening
response.onInterrupt(eventId, "realtime", null);
response.onLlmItemDelta(eventId, itemId, "Mhm, I heard you...");

// Risk-control hit (interruptTips is mandatory, used as the spoken reply)
response.onInterrupt(eventId, "risk_control",
    "You mentioned a sensitive topic; let's talk about something else");

Shared Base Callbacks (v1.4.0+)

PassiveCallback declares a set of "shared" callbacks that are independent of the specific semantic mode. Every one of the 8 callback classes (Audio2Tts*, Asr2Llm*, ...) inherits them, so just override the ones you care about:

MethodTriggerNotes
onRobotOnline(agentId, agentMeta)Agent onlineagentId is the agent ID configured on the LinkSoul platform; agentMeta carries device metadata (wake word, city, etc.)
onRobotOffline(agentId)Agent offlineRelease resources associated with this agentId
onFaceInfo(agentId, eventId, param)Face / voiceprint UID detectedparam carries face or voiceprint fields
onVideoFrame(agentId, eventId, flag, buf, isKeyFrame, localTs, param)Video stream passthroughflag=2 H264, flag=3 image
onGreetSignal(agentId, eventId, param, GreetResponse response)Active-greeting signalReply via VLM or TTS through response
onState(agentId, eventId, stateName, stateValue)Robot state changestateValue may be a JSON string; see robot-side state modules for every stateName

v1.4.0 change: the previously separate Greet2VlmCallback and StateListenCallback have been merged into PassiveCallback. There is no separate register* for them anymore — just override these methods on whichever business callback (e.g., Audio2TtsCallback) you register.

Greeting reply (GreetResponse)

After onGreetSignal, you can reply on GreetResponse in either of two modes (not mutually exclusive — pick whichever applies):

java
@Override
public void onGreetSignal(String agentId, String eventId, AgentParam param, GreetResponse response) {
    // Mode 1: streamed VLM text
    response.onGreetVlmDelta(eventId, "Hello");
    response.onGreetVlmDelta(eventId, ", nice weather today");
    response.onGreetVlmDone(eventId);

    // Mode 2: streamed TTS audio — argument is a base64-encoded chunk of synthesised audio bytes
    byte[] audioChunk1 = synthesizeTts("Hello");                  // PCM / Opus / ... bytes
    byte[] audioChunk2 = synthesizeTts(", welcome home");
    response.onGreetTtsDelta(eventId, Base64.getEncoder().encodeToString(audioChunk1));
    response.onGreetTtsDelta(eventId, Base64.getEncoder().encodeToString(audioChunk2));
    response.onGreetTtsDone(eventId);

    // Error path
    // response.onError(eventId, 5101, "greeting generation failed");
}
MethodDescription
onGreetVlmDelta(eventId, text)VLM streaming text chunk
onGreetVlmDone(eventId)VLM streaming finished
onGreetTtsDelta(eventId, audioBase64)TTS audio chunk — audioBase64 is the base64-encoded chunk of synthesised audio bytes
onGreetTtsDone(eventId)TTS audio stream finished
onError(eventId, code, msg)Error reply

Video frame passthrough

onVideoFrame covers both the video data feeding Asr2* / AudioVideo2* callbacks and plain video passthrough scenarios:

flagMeaningbuf
2H264 frameH264 bytes; isKeyFrame / localTs valid
3Image frameJPEG/PNG bytes; isKeyFrame=false, localTs=-1

Important Notes

  1. Callback is a singleton: every online agent shares the same callback instance; use agentId (the agent ID configured on the LinkSoul platform) to distinguish them
  2. Threading model: each agentId is bound to a dedicated thread; requests from the same agent are executed sequentially
  3. eventId isolation: Different conversation rounds are isolated by eventId; non-thread-safe resources should be managed per eventId
  4. Response is thread-safe: response.on*() methods can be called from any thread
  5. Response timeout cleanup: Response objects are automatically cleaned up after 120 seconds of inactivity; long-running tasks must call response methods periodically to stay active
  6. Register before initialize: You must call register*Callback() before initialize(); the order cannot be reversed
  7. Override shared callbacks as needed: onGreetSignal / onState / onFaceInfo / onVideoFrame ship with default no-op (log-only) implementations on PassiveCallback, so unused ones can be left alone