Passive Interaction Examples

Source location: agentsdk_for_java/example/src/main/java/com/agibot/aiem/

The 8 passive examples are grouped into three categories by input/output complexity. Every *Example.java file is a self-contained main; only the register* method and the onRequest signature differ. Everything else — SDK creation, auth registration, shared base callbacks, initialization — is identical.

Shared skeleton (used by every example)

java

String url = "wss://open.agibot.com/api/V1/open-portal/app/wss/agent-sdk";
String appId = "<your appId from the LinkSoul open platform>";
String appKey = "<your appKey from the LinkSoul open platform>";
String appSecret = "<your appSecret from the LinkSoul open platform>";

// Step 1: create the SDK instance (singleton per appId)
AgentSdk agentSdk = AgentSdk.create(url, appId, appKey, appSecret);

// Step 2: register the auth callback
agentSdk.registerAuth(new AgentAuthCallback() {
    @Override
    public void onAuthState(String appId, int code, String msg) {
        log.info("onAuthState => appId: {}, code: {}, msg: {}", appId, code, msg);
    }
});

// Step 3: register your business callback (see each example below)
agentSdk.registerXxx(new XxxCallback(agentSdk) { /* ... */ });

// Step 4: initialize (opens the WebSocket, auto-reconnect)
agentSdk.initialize();

Shared base callbacks (v1.4.0+)

Every *Callback extends PassiveCallback and may optionally override the following methods (they have safe no-op log defaults):

java

@Override
public void onRobotOnline(String agentId, AgentMeta agentMeta) {
    // Robot online — initialize per-robot resources
    log.info("onRobotOnline => agentId: {}, agentMeta: {}", agentId, agentMeta);
}

@Override
public void onRobotOffline(String agentId) {
    // Robot offline — release per-robot resources
    log.info("onRobotOffline => agentId: {}", agentId);
}

@Override
public void onFaceInfo(String agentId, String eventId, AgentParam param) {
    // Face / voiceprint UID detected (opened in v1.4.0)
    log.info("onFaceInfo => agentId {}, eventId {}, param {}", agentId, eventId, param);
}

@Override
public void onVideoFrame(String agentId, String eventId, int flag, byte[] buf,
                         boolean isKeyFrame, long localTs, AgentParam param) {
    // Video passthrough: flag=2 H264, flag=3 image
    log.info("onVideoFrame => agentId {}, eventId {}, flag {}, isKeyFrame {}, localTs {}, param {}",
        agentId, eventId, flag, isKeyFrame, localTs, param);
}

@Override
public void onGreetSignal(String agentId, String eventId, AgentParam param, GreetResponse response) {
    // Greeting signal; reply via response.onGreetVlm*/onGreetTts*
    log.info("onGreetSignal => agentId {}, eventId {}, param {}", agentId, eventId, param);
}

@Override
public void onState(String agentId, String eventId, String stateName, String stateValue) {
    // Robot state push (merged here in v1.4.0; no more separate StateListenCallback)
    log.info("onState => agentId {}, eventId {}, stateName {}, stateValue {}",
        agentId, eventId, stateName, stateValue);
}

Each example below shows only the register* call and the body of onRequest. Wrap it with the shared skeleton above to get a runnable program.

1. Pure audio

Audio2LlmExample — audio in → ASR + LLM text out

Source: Audio2LlmExample.java
Scenario: voice conversation with text-only reply (no TTS).

java

agentSdk.registerAudio2Llm(new Audio2LlmCallback(agentSdk) {

    /**
     * @param flag  0 VAD start, 1 audio chunk, 2 VAD end
     * @param buf   audio bytes (valid when flag == 1)
     */
    @Override
    public void onRequest(String agentId, String eventId, int flag,
                          byte[] buf, AgentParam param, Audio2LlmResponse response) {

        // Optional: agent metadata
        AgentMeta agentMeta = agentSdk.getAgentMeta(agentId);
        String defWakeupWord = agentMeta.getWakeupWord();
        String yourParamKey = agentMeta.getString("your_custom_key");

        // 1) ASR by flag
        if (flag == 0) {
            // init ASR client
        } else if (flag == 1) {
            // feed audio chunk to ASR
        } else if (flag == 2) {
            // VAD end
        }
        response.onAsrMiddle(eventId, "today");
        response.onAsrFinal(eventId, "what's the weather like today");

        // 2) 语义理解 / LLM streaming
        String itemId = "itemId_000001";

        // Optional barge-in
        response.onInterrupt(eventId, "chat", null);

        // Optional skill dispatch
        AgentParam skillParam = AgentParam.create().setInteger("step", 3);
        response.onSkill(eventId, itemId, "movement", "move_forward", skillParam);

        String textDelta = "the LLM streaming text goes here";
        response.onLlmItemDelta(eventId, itemId, textDelta);
        response.onLlmItemDone(eventId, itemId);
        response.onLlmDone(eventId);

        // Error handling
        // response.onError(eventId, 1010, "ASR call failed");
    }

    // ... shared base callbacks ...
});

Audio2TtsExample — audio in → ASR + LLM + TTS full pipeline

Source: Audio2TtsExample.java
Scenario: most common voice-in → voice-out conversational loop. The onRequest signature carries an extra itemId compared to Audio2Llm.

java

agentSdk.registerAudio2Tts(new Audio2TtsCallback(agentSdk) {

    /**
     * @param itemId for multi-intent replies
     * @param flag   0 VAD start, 1 audio chunk, 2 VAD end
     * @param buf    audio bytes (flag == 1)
     */
    @Override
    public void onRequest(String agentId, String eventId, String itemId, int flag,
                          byte[] buf, AgentParam param, Audio2TtsResponse response) {

        // 1) ASR
        if (flag == 0) {
            // init ASR
        } else if (flag == 1) {
            // feed audio
        } else if (flag == 2) {
            // VAD end
        }
        response.onAsrMiddle(eventId, "today");
        response.onAsrFinal(eventId, "what's the weather like today");

        // 2) LLM streaming
        String textDelta = "the LLM streaming text goes here";
        response.onLlmItemDelta(eventId, itemId, textDelta);
        response.onLlmItemDone(eventId, itemId);
        response.onLlmDone(eventId);

        // 3) Optional barge-in + skill
        response.onInterrupt(eventId, "chat", null);
        AgentParam skillParam = AgentParam.create().setInteger("step", 3);
        response.onSkill(eventId, itemId, "movement", "move_forward", skillParam);

        // 4) TTS streaming
        byte[] audio = new byte[1024];
        response.onTtsItemDelta(eventId, itemId, audio);
        response.onTtsItemDone(eventId, itemId);
        response.onTtsDone(eventId);

        // Error handling
        // response.onError(eventId, 1010, "ASR call failed");
    }

    // ... shared base callbacks ...
});

2. Pure text

Asr2LlmExample — ASR text in → LLM out

Source: Asr2LlmExample.java
Scenario: ASR done upstream; you only need 语义理解/LLM.

java

agentSdk.registerAsr2Llm(new Asr2LlmCallback(agentSdk) {

    @Override
    public void onRequest(String agentId, String eventId, String text,
                          AgentParam param, Asr2LlmResponse response) {

        // Run 语义理解 + LLM on `text`
        response.onInterrupt(eventId, "chat", null);

        String itemId = "itemId_000001";
        AgentParam skillParam = AgentParam.create().setInteger("step", 3);
        response.onSkill(eventId, itemId, "movement", "move_forward", skillParam);

        String textDelta = "the LLM streaming text goes here";
        response.onLlmItemDelta(eventId, itemId, textDelta);
        response.onLlmItemDone(eventId, itemId);
        response.onLlmDone(eventId);

        // response.onError(eventId, 1010, "...");
    }

    // ... shared base callbacks ...
});

Asr2TtsExample — ASR text in → LLM + TTS out

Source: Asr2TtsExample.java
Scenario: ASR done upstream; downstream needs voice reply.

java

agentSdk.registerAsr2Tts(new Asr2TtsCallback(agentSdk) {

    @Override
    public void onRequest(String agentId, String eventId, String text,
                          AgentParam param, Asr2TtsResponse response) {

        String itemId = "itemId_000001";

        // LLM
        String textDelta = "the LLM streaming text goes here";
        response.onLlmItemDelta(eventId, itemId, textDelta);
        response.onLlmItemDone(eventId, itemId);
        response.onLlmDone(eventId);

        // Barge-in + skill
        response.onInterrupt(eventId, "chat", null);
        AgentParam skillParam = AgentParam.create().setInteger("step", 3);
        response.onSkill(eventId, itemId, "movement", "move_forward", skillParam);

        // TTS
        byte[] audio = new byte[1024];
        response.onTtsItemDelta(eventId, itemId, audio);
        response.onTtsItemDone(eventId, itemId);
        response.onTtsDone(eventId);
    }

    // ... shared base callbacks ...
});

3. Multimodal (text/audio + video)

AsrVideo2VlmExample — ASR text + video → VLM out

Source: AsrVideo2VlmExample.java
Scenario: visual question answering.

java

agentSdk.registerAsrVideo2Vlm(new AsrVideo2VlmCallback(agentSdk) {

    /**
     * @param flag  3 ASR text, 4 H264, 5 image
     * @param text  valid when flag == 3
     * @param buf   valid when flag == 4 or 5
     */
    @Override
    public void onRequest(String agentId, String eventId, int flag, String text,
                          byte[] buf, AgentParam param, AsrVideo2VlmResponse response) {

        // Run VLM on `text` plus the buffered video frame
        String itemId = "itemId_000001";

        response.onInterrupt(eventId, "chat", null);
        AgentParam skillParam = AgentParam.create().setInteger("step", 3);
        response.onSkill(eventId, itemId, "movement", "move_forward", skillParam);

        String textDelta = "the LLM streaming text goes here";
        response.onVlmItemDelta(eventId, itemId, textDelta);
        response.onVlmItemDone(eventId, itemId);
        response.onVlmDone(eventId);
    }

    // ... shared base callbacks ...
});

AsrVideo2TtsExample — ASR text + video → VLM + TTS out

Source: AsrVideo2TtsExample.java
Scenario: visual QA with spoken reply.

java

agentSdk.registerAsrVideo2Tts(new AsrVideo2TtsCallback(agentSdk) {

    @Override
    public void onRequest(String agentId, String eventId, int flag, String text,
                          byte[] buf, AgentParam param, AsrVideo2TtsResponse response) {

        String itemId = "itemId_000001";
        String textDelta = "the LLM streaming text goes here";

        // 1) VLM
        response.onVlmItemDelta(eventId, itemId, textDelta);
        response.onVlmItemDone(eventId, itemId);
        response.onVlmDone(eventId);

        // 2) Barge-in + skill
        response.onInterrupt(eventId, "chat", null);
        AgentParam skillParam = AgentParam.create().setInteger("step", 3);
        response.onSkill(eventId, itemId, "movement", "move_forward", skillParam);

        // 3) TTS
        byte[] audio = new byte[1024];
        response.onTtsItemDelta(eventId, itemId, audio);
        response.onTtsItemDone(eventId, itemId);
        response.onTtsDone(eventId);
    }

    // ... shared base callbacks ...
});

AudioVideo2VlmExample — audio + video → ASR + VLM out

Source: AudioVideo2VlmExample.java
Scenario: full multimodal understanding, text reply.

java

agentSdk.registerAudioVideo2Vlm(new AudioVideo2VlmCallback(agentSdk) {

    /**
     * @param flag  0 VAD start, 1 audio chunk, 2 VAD end, 3 H264 video frame
     * @param buf   audio bytes (flag=1) or H264 bytes (flag=3)
     */
    @Override
    public void onRequest(String agentId, String eventId, int flag, byte[] buf,
                          AgentParam param, AudioVideo2VlmResponse response) {

        // 1) ASR
        if (flag == 0) {
            // init ASR
        } else if (flag == 1) {
            // feed audio
        } else if (flag == 2) {
            // VAD end
        }
        response.onAsrMiddle(eventId, "today");
        response.onAsrFinal(eventId, "what's the weather like today");

        String itemId = "itemId_000001";

        // 2) Barge-in + skill
        response.onInterrupt(eventId, "chat", null);
        AgentParam skillParam = AgentParam.create().setInteger("step", 3);
        response.onSkill(eventId, itemId, "movement", "move_forward", skillParam);

        // 3) VLM streaming (combined with video frames)
        String textDelta = "the LLM streaming text goes here";
        response.onVlmItemDelta(eventId, itemId, textDelta);
        response.onVlmItemDone(eventId, itemId);
        response.onVlmDone(eventId);
    }

    // ... shared base callbacks ...
});

AudioVideo2TtsExample — audio + video → ASR + VLM + TTS out

Source: AudioVideo2TtsExample.java
Scenario: full multimodal end-to-end — closest to real robot conversations.

java

agentSdk.registerAudioVideo2Tts(new AudioVideo2TtsCallback(agentSdk) {

    /**
     * @param flag  0 VAD start, 1 audio chunk, 2 VAD end, 3 H264 video frame
     * @param buf   audio bytes (flag=1) or H264 bytes (flag=3)
     */
    @Override
    public void onRequest(String agentId, String eventId, int flag, byte[] buf,
                          AgentParam param, AudioVideo2TtsResponse response) {

        // 1) ASR
        if (flag == 0) {
            // init ASR
        } else if (flag == 1) {
            // feed audio
        } else if (flag == 2) {
            // VAD end
        }
        response.onAsrMiddle(eventId, "today");
        response.onAsrFinal(eventId, "what's the weather like today");

        String itemId = "itemId_000001";

        // 2) Barge-in + skill
        response.onInterrupt(eventId, "chat", null);
        AgentParam skillParam = AgentParam.create().setInteger("step", 3);
        response.onSkill(eventId, itemId, "movement", "move_forward", skillParam);

        // 3) VLM streaming
        String textDelta = "the LLM streaming text goes here";
        response.onVlmItemDelta(eventId, itemId, textDelta);
        response.onVlmItemDone(eventId, itemId);
        response.onVlmDone(eventId);

        // 4) TTS streaming
        byte[] audio = new byte[1024];
        response.onTtsItemDelta(eventId, itemId, audio);
        response.onTtsItemDone(eventId, itemId);
        response.onTtsDone(eventId);
    }

    // ... shared base callbacks ...
});

Cross-cutting concerns

Skill dispatch

When the LLM/VLM decides a skill (navigation, grasping, ...) should fire:

java

AgentParam skillParam = AgentParam.create()
    .setInteger("step", 3)
    .setDouble("speed", 0.5)
    .setString("target", "kitchen");

response.onSkill(eventId, itemId, "movement", "walk_forward", skillParam);

Error handling

Report failures back to the robot with response.onError(...):

java

try {
    // business logic ...
} catch (AsrServiceException e) {
    response.onError(eventId, 5001, "ASR service unavailable: " + e.getMessage());
} catch (LlmServiceException e) {
    response.onError(eventId, 5002, "LLM service timeout");
} catch (Exception e) {
    response.onError(eventId, 5999, "unknown error");
}

Which example to pick?

Your input → output	Recommended example
audio → text	`Audio2LlmExample`
audio → speech	`Audio2TtsExample`
text → text	`Asr2LlmExample`
text → speech	`Asr2TtsExample`
text + video → text	`AsrVideo2VlmExample`
text + video → speech	`AsrVideo2TtsExample`
audio + video → text	`AudioVideo2VlmExample`
audio + video → speech	`AudioVideo2TtsExample`