Passive Interaction Development Guide
Passive Interaction Development Guide
Passive interaction is the core functionality of the SDK: Robot initiates a request (audio/video/text) -> SDK callback processes it -> Returns result via Response.
8 Passive Callback Types
Choose the appropriate callback type based on input/output combination. Each SDK instance can register only one passive callback.
Audio-Only
| Callback Class | Input | Output | Typical Scenario |
|---|---|---|---|
Audio2LlmCallback | Audio stream | ASR + LLM text | Voice conversation, text reply |
Audio2TtsCallback | Audio stream | ASR + LLM + TTS audio | Voice conversation, voice reply |
Text-Only
| Callback Class | Input | Output | Typical Scenario |
|---|---|---|---|
Asr2LlmCallback | ASR text | LLM text | Text conversation |
Asr2TtsCallback | ASR text | LLM + TTS audio | Text to voice reply |
Multimodal (Audio+Video / Text+Video)
| Callback Class | Input | Output | Typical Scenario |
|---|---|---|---|
AsrVideo2VlmCallback | ASR text + Video | VLM text | Image understanding |
AsrVideo2TtsCallback | ASR text + Video | VLM + TTS audio | Image understanding with voice reply |
AudioVideo2VlmCallback | Audio + Video | ASR + VLM text | Full multimodal understanding |
AudioVideo2TtsCallback | Audio + Video | ASR + VLM + TTS | Full multimodal voice reply |
v1.4.0: the
Audio2Asr/Audio2Audio/AudioVideo2Audio/Text2Ttscallbacks and their Response classes have been removed from the SDK.
Flag Values Explained
flag is the key parameter for distinguishing different data types within the same callback.
Audio Callbacks (Audio2*, AudioVideo2*)
| flag | Meaning | buf Content | Trigger Condition |
|---|---|---|---|
0 | VAD start | null | User starts speaking |
1 | Audio frame | PCM audio data | Continuous transmission |
2 | VAD end | null | User stops speaking |
4 | H264 video frame | H264 encoded data | Video stream transmission |
5 | Image frame | JPEG/PNG image | Image transmission |
ASR+Video Callbacks (AsrVideo2*)
| flag | Meaning | Valid Parameters | Trigger Condition |
|---|---|---|---|
3 | ASR text | text is valid, buf=null | ASR recognition complete |
4 | H264 video frame | buf is valid, text=null | Video stream transmission |
5 | Image frame | buf is valid, text=null | Image transmission |
Key Understanding: The
onRequestmethod ofAsrVideo2*callbacks will be called multiple times (separately delivering text and video). You need to check theflagvalue to determine what data is being passed in each call.
Development Template
agentSdk.registerAudio2Tts(new Audio2TtsCallback(agentSdk) {
@Override
public void onRobotOnline(String agentId, AgentMeta agentMeta) {
// Robot comes online, initialize resources for this robot
// agentMeta contains wakeup word, city, and other configuration
String wakeupWord = agentMeta.getWakeupWord();
}
@Override
public void onRobotOffline(String agentId) {
// Robot goes offline, clean up resources for this robot
}
@Override
public void onRequest(String agentId, String eventId, String itemId,
int flag, byte[] audio, AgentParam param,
Audio2TtsResponse response) {
switch (flag) {
case 0 -> {
// VAD start: initialize ASR session
}
case 1 -> {
// Audio frame: send to ASR service for streaming recognition
sendToAsr(audio);
}
case 2 -> {
// VAD end: finalize ASR, call LLM, synthesize TTS
String asrText = finalizeAsr();
response.onAsrFinal(eventId, asrText);
// Call LLM to get reply
String llmReply = callLlm(asrText);
response.onLlmItemDelta(eventId, itemId, llmReply);
response.onLlmItemDone(eventId, itemId);
response.onLlmDone(eventId);
// Synthesize TTS audio
byte[] ttsAudio = synthesizeTts(llmReply);
response.onTtsItemDelta(eventId, itemId, ttsAudio);
response.onTtsItemDone(eventId, itemId);
response.onTtsDone(eventId);
}
}
}
});
Response Methods Overview
All Response classes inherit from AgentSdkResponse. Available method combinations differ by callback type:
| Method | Description | Parameters |
|---|---|---|
onAsrMiddle(eventId, text) | ASR intermediate result (streaming) | text: currently recognized text |
onAsrFinal(eventId, text) | ASR final result | text: complete recognized text |
onLlmItemDelta(eventId, itemId, text) | LLM streaming output | text: incremental text for this chunk |
onLlmItemDone(eventId, itemId) | LLM single segment complete | |
onLlmDone(eventId) | LLM all segments complete | |
onVlmItemDelta(eventId, itemId, text) | VLM streaming output | text: incremental text for this chunk |
onVlmItemDone(eventId, itemId) | VLM single segment complete | |
onVlmDone(eventId) | VLM all segments complete | |
onTtsItemDelta(eventId, itemId, audio) | TTS audio chunk | audio: PCM byte array |
onTtsItemDone(eventId, itemId) | TTS single segment complete | |
onTtsDone(eventId) | TTS all segments complete | |
onSkill(eventId, itemId, skillType, skillName, param) | Dispatch skill command | param: AgentParam; see Semantic Skills for the full catalogue |
onInterrupt(eventId, type, tips) | Interrupt current conversation | type ∈ risk_control / action / chat / realtime; tips required only for risk_control. See Interrupts |
onError(eventId, errorCode, errorMsg) | Return error | Supported by all Response types |
Response Available Methods Matrix
| Response Class | ASR | LLM | VLM | TTS | Skill | Interrupt |
|---|---|---|---|---|---|---|
Audio2LlmResponse | ✓ | ✓ | ✓ | ✓ | ||
Audio2TtsResponse | ✓ | ✓ | ✓ | ✓ | ✓ | |
Asr2LlmResponse | ✓ | ✓ | ✓ | |||
Asr2TtsResponse | ✓ | ✓ | ✓ | ✓ | ||
AsrVideo2VlmResponse | ✓ | ✓ | ✓ | |||
AsrVideo2TtsResponse | ✓ | ✓ | ✓ | ✓ | ||
AudioVideo2VlmResponse | ✓ | ✓ | ✓ | ✓ | ||
AudioVideo2TtsResponse | ✓ | ✓ | ✓ | ✓ | ✓ |
Standard Response Flow
onRequest(flag=0) -> Initialize
onRequest(flag=1) -> Accumulate audio -> ASR streaming recognition
-> onAsrMiddle() (optional, intermediate result)
onRequest(flag=2) -> ASR complete -> onAsrFinal()
-> LLM call -> onLlmItemDelta() x N
-> onLlmItemDone()
-> onLlmDone()
-> TTS synthesis -> onTtsItemDelta() x N
-> onTtsItemDone()
-> onTtsDone()
When to dispatch a skill
Trigger: once you have an ASR text, run 语义理解 (intent classification) on it. If the intent maps
to a skill (movement, action, expression, ...), dispatch it through response.onSkill(...).
If it does not, just continue with the normal LLM/VLM text reply — do not call onSkill.
There are two ways the ASR text arrives:
| Callback type | Where the text comes from | When to classify |
|---|---|---|
Asr2Llm / Asr2Tts / AsrVideo2Vlm / AsrVideo2Tts | Upstream (the robot side) already finished ASR and hands you the final text directly through onRequest(... text ...) | As soon as text arrives |
Audio2Llm / Audio2Tts / AudioVideo2Vlm / AudioVideo2Tts | The SDK side accumulates flag=1 audio frames and produces the final text at flag=2 (VAD end) | After computing the final text — typically right after response.onAsrFinal in the flag=2 branch |
Typical flow (Asr2Llm shown):
onRequest(text="please walk forward 10 meters")
│
▼
语义理解 classification
├─ rejected (invalid query / false wake / noise / ...)
│ └→ just return — do NOT call any response.on* method
│ (no skill, no interrupt, no LLM/VLM output)
│
└─ valid intent
① first MUST call response.onInterrupt(eventId, type, tips)
│ (type ∈ risk_control / action / chat / realtime — see "Interrupts" below)
│
② then fill in the result:
├─ skill hit → response.onSkill(eventId, itemId, ...)
├─ valid chat → response.onLlmItemDelta / onLlmItemDone / onLlmDone
├─ multimodal → response.onVlmItemDelta / onVlmItemDone / onVlmDone
└─ TTS reply (when applicable) → response.onTtsItemDelta / onTtsItemDone / onTtsDone
⚠️ Strict ordering: once the intent is classified as valid, you MUST call
onInterruptfirst, then any ofonSkill/onLlm*/onVlm*/onTts*. Reversing the order leaves the robot stuck on the previous reply. The only exception is rejected — call nothing, let the robot keep its current state.
Example (Asr2Llm; the Audio2Tts case does the same 语义理解 step inside its flag=2 branch):
@Override
public void onRequest(String agentId, String eventId, String text,
AgentParam param, Asr2LlmResponse response) {
// 语义理解 on the business side: classify `text` into
// "skill / chat / realtime / risk_control / rejected"
NluResult nlu = classify(text);
// ① Rejected: just return — do not call any response.on* method
if (nlu.isRejected()) {
return;
}
String itemId = IdGenerator.generateItemId();
// ② Valid intent — MUST interrupt first, then fill in skill / LLM / VLM / TTS
if (nlu.isRiskControl()) {
// Risk control: interruptTips is required
response.onInterrupt(eventId, "risk_control",
"You mentioned a sensitive topic; let's talk about something else");
return; // typically no further skill / LLM output after risk control
} else if (nlu.isSkill()) {
response.onInterrupt(eventId, "action", null); // skill interrupt
} else {
response.onInterrupt(eventId, "chat", null); // valid-chat interrupt
// For realtime dialog: response.onInterrupt(eventId, "realtime", null);
}
// ③ Skill hit: dispatch skillParam (keys/values come from the Semantic Skills reference)
if (nlu.isSkill()) {
AgentParam skillParam = AgentParam.create()
.setString("movement", "move_forward")
.setInteger("meter", 10);
response.onSkill(eventId, itemId, "movement", "move_forward", skillParam);
}
// ④ LLM streaming reply (runs alongside the skill — they do not block each other)
response.onLlmItemDelta(eventId, itemId, "Sure, walking forward.");
response.onLlmItemDone(eventId, itemId);
response.onLlmDone(eventId);
}
skillType/skillNameand the contents ofskillParamare listed in Semantic Skills. MultipleonSkillcalls under the sameeventIdare allowed (multi-intent scenarios) — use a distinctitemIdper intent.
Interrupts (onInterrupt)
response.onInterrupt(eventId, interruptType, interruptTips) is called before a new reply so
that the robot stops whatever it is currently doing (playback / motion / conversation) before the
new TTS / skill / dialog content starts.
interruptType values
| Value | Meaning | Scenario | interruptTips |
|---|---|---|---|
risk_control | Risk-control interrupt | Content moderation, sensitive topic; the current dialog must be cut immediately | Required, e.g. "You mentioned a sensitive topic; let's talk about something else" |
action | Skill interrupt | 语义理解 matched a skill / action — barge in on the current motion or playback before dispatching the new action | Pass null |
chat | Valid-chat interrupt | 语义理解 matched plain chat (valid conversation) — a new LLM/VLM text reply starts and cuts off the old one | Pass null |
realtime | Realtime interrupt | Talk-while-listening (realtime) dialog — a new realtime segment cuts off the old one | Pass null |
interruptTipsis a user-facing message. Onlyrisk_controlrequires it — the robot will speak that line to the user when the risk-control interrupt fires. For the other three types, passnullby convention.
Call order (mandatory)
⚠️ For every valid intent, the response sequence below is mandatory, otherwise the robot cannot enter the new reply state:
response.onInterrupt(eventId, type, tips)response.onSkill(...)and / orresponse.onLlmItem* / onVlmItem* / onTtsItem*
Pick the interruptType based on the 语义理解 classification:
| 语义理解 result | First call | Allowed subsequent replies |
|---|---|---|
| Risk-control hit | onInterrupt(..., "risk_control", "<spoken prompt>") | Typically no further LLM/VLM/skill replies (business-dependent) |
| Skill hit | onInterrupt(..., "action", null) | onSkill(...) (can run in parallel with onLlm* / onVlm* / onTts*) |
| Valid chat | onInterrupt(..., "chat", null) | onLlmItem* / onVlmItem* / onTtsItem* |
| Realtime dialog | onInterrupt(..., "realtime", null) | Realtime reply content |
| Rejected | Do not call onInterrupt | Call nothing — let the robot keep its current state |
Examples
// Plain chat reply
response.onInterrupt(eventId, "chat", null);
response.onLlmItemDelta(eventId, itemId, "Sure, here is the answer.");
// Action / skill hit
response.onInterrupt(eventId, "action", null);
AgentParam skillParam = AgentParam.create().setString("action", "wave_hands");
response.onSkill(eventId, itemId, "action", "wave_hands", skillParam);
// Realtime talk-while-listening
response.onInterrupt(eventId, "realtime", null);
response.onLlmItemDelta(eventId, itemId, "Mhm, I heard you...");
// Risk-control hit (interruptTips is mandatory, used as the spoken reply)
response.onInterrupt(eventId, "risk_control",
"You mentioned a sensitive topic; let's talk about something else");
Shared Base Callbacks (v1.4.0+)
PassiveCallback declares a set of "shared" callbacks that are independent of the specific
semantic mode. Every one of the 8 callback classes (Audio2Tts*, Asr2Llm*, ...) inherits
them, so just override the ones you care about:
| Method | Trigger | Notes |
|---|---|---|
onRobotOnline(agentId, agentMeta) | Agent online | agentId is the agent ID configured on the LinkSoul platform; agentMeta carries device metadata (wake word, city, etc.) |
onRobotOffline(agentId) | Agent offline | Release resources associated with this agentId |
onFaceInfo(agentId, eventId, param) | Face / voiceprint UID detected | param carries face or voiceprint fields |
onVideoFrame(agentId, eventId, flag, buf, isKeyFrame, localTs, param) | Video stream passthrough | flag=2 H264, flag=3 image |
onGreetSignal(agentId, eventId, param, GreetResponse response) | Active-greeting signal | Reply via VLM or TTS through response |
onState(agentId, eventId, stateName, stateValue) | Robot state change | stateValue may be a JSON string; see robot-side state modules for every stateName |
v1.4.0 change: the previously separate
Greet2VlmCallbackandStateListenCallbackhave been merged intoPassiveCallback. There is no separateregister*for them anymore — just override these methods on whichever business callback (e.g.,Audio2TtsCallback) you register.
Greeting reply (GreetResponse)
After onGreetSignal, you can reply on GreetResponse in either of two modes
(not mutually exclusive — pick whichever applies):
@Override
public void onGreetSignal(String agentId, String eventId, AgentParam param, GreetResponse response) {
// Mode 1: streamed VLM text
response.onGreetVlmDelta(eventId, "Hello");
response.onGreetVlmDelta(eventId, ", nice weather today");
response.onGreetVlmDone(eventId);
// Mode 2: streamed TTS audio — argument is a base64-encoded chunk of synthesised audio bytes
byte[] audioChunk1 = synthesizeTts("Hello"); // PCM / Opus / ... bytes
byte[] audioChunk2 = synthesizeTts(", welcome home");
response.onGreetTtsDelta(eventId, Base64.getEncoder().encodeToString(audioChunk1));
response.onGreetTtsDelta(eventId, Base64.getEncoder().encodeToString(audioChunk2));
response.onGreetTtsDone(eventId);
// Error path
// response.onError(eventId, 5101, "greeting generation failed");
}
| Method | Description |
|---|---|
onGreetVlmDelta(eventId, text) | VLM streaming text chunk |
onGreetVlmDone(eventId) | VLM streaming finished |
onGreetTtsDelta(eventId, audioBase64) | TTS audio chunk — audioBase64 is the base64-encoded chunk of synthesised audio bytes |
onGreetTtsDone(eventId) | TTS audio stream finished |
onError(eventId, code, msg) | Error reply |
Video frame passthrough
onVideoFrame covers both the video data feeding Asr2* / AudioVideo2* callbacks and
plain video passthrough scenarios:
| flag | Meaning | buf |
|---|---|---|
2 | H264 frame | H264 bytes; isKeyFrame / localTs valid |
3 | Image frame | JPEG/PNG bytes; isKeyFrame=false, localTs=-1 |
Important Notes
- Callback is a singleton: every online agent shares the same callback instance; use
agentId(the agent ID configured on the LinkSoul platform) to distinguish them - Threading model: each
agentIdis bound to a dedicated thread; requests from the same agent are executed sequentially - eventId isolation: Different conversation rounds are isolated by
eventId; non-thread-safe resources should be managed pereventId - Response is thread-safe:
response.on*()methods can be called from any thread - Response timeout cleanup: Response objects are automatically cleaned up after 120 seconds of inactivity; long-running tasks must call response methods periodically to stay active
- Register before initialize: You must call
register*Callback()beforeinitialize(); the order cannot be reversed - Override shared callbacks as needed:
onGreetSignal/onState/onFaceInfo/onVideoFrameship with default no-op (log-only) implementations onPassiveCallback, so unused ones can be left alone