8. Guide to using interaction secondary development

8. Guide to using interaction secondary development

For client-side secondary development related to interaction, two development modes are provided:

  1. Simple extension of A2 interaction
    • Agibot retains its core interaction capabilities and provides fundamental interfaces for TTS, gestures, and facial expressions. These interfaces allow users to control the robot for basic workflows such as exhibition guiding and presentations, while the robot continues to fully support Agibot's standard voice interaction solution.
  2. Full takeover for secondary development
    • Disable Agibot's interaction capabilities. Agibot provides the robot's microphone input (after onboard noise reduction), allowing the secondary development program to fully take over all voice interaction tasks.

The following table provides a detailed comparison of the two.

ModeAgibot Interaction Retained?Interfaces Provided by AgibotInternet Connectivity RequirementsApplicable ScenariosApplicable Scenarios
Simple Extension of Agibot InteractionYesTTS playback, Speaker playbackBoth the Agibot interaction solution and TTS interfaces require an internet connection.Developing simple exhibition guide demos with fixed workflow orchestration.Relatively simple. Workload depends on workflow complexity but is generally low.
Full Takeover by Secondary DevelopmentNoNoise-cancelled audio, SpeakerInternet is required for robot initialization, but not for subsequent usage.Developing a custom, complete interaction solution.High difficulty. Requires a complete R&D team; significant workload.

8.1 Simple extension of A2 interaction

In this mode, interfaces provided by Agibot—such as expression control, action playback, TTS playback, and face recognition—can be used to implement specific functions. However, please note that most of these interfaces are not yet available. Detailed instructions will be provided once these interfaces are officially released.

8.2 Full control for secondary development

Interaction-based secondary development involves three key components: disabling Agibot's interaction capabilities, acquiring microphone audio, and utilizing the speaker. The following sections will detail how to proceed with secondary development across these three areas.

8.2.1 Disable the A2 interaction capability

To enable full takeover by secondary development, you must first disable Agibot's interaction capabilities so that only the microphone audio (including onboard noise reduction) is output. Please follow the steps below: Adjust the agent module to only_voice mode. You can use the SetAgentPropertiesRequest interface in the Microphone Management section to achieve this.

  • only_voice: Outputs only the noise-cancelled microphone audio to /agent/process_audio_output. All downstream processing chains are disconnected.
    The RPC calls required are as follows:
    bash
    curl -i \
        -H 'content-type:application/json' \
        -X POST 'http://192.168.100.110:59301/rpc/aimdk.protocol.AgentControlService/SetAgentPropertiesRequest' \
        -d '{ "contents": { "properties": { "2": "only_voice" } } }'
    

Please note that a robot restart is required for the changes to take effect. You can wait until you have completed the modifications in Step 2 and then restart the robot all at once. After the restart, you can verify the current interaction mode by using the GetAgentPropertiesRequest interface in the Microphone Management section to confirm if the change was successful. (To restore the default settings, simply change "only_voice" to "normal" in the SetAgentPropertiesRequest interface.))

8.2.2 Microphone Audio Acquisition

Note: To obtain the audio data described below, the robot must be connected to the internet for at least 2 minutes upon startup to complete audio-related authentication. Otherwise, no raw audio will be output. If you intend to use it offline, please ensure that this interface is outputting audio successfully before disconnecting from the network. After exiting the interaction loop, you can obtain the noise-reduced microphone audio from the robot via the /agent/process_audio_output topic. An example program for retrieving this audio is provided below:

python
#!/usr/bin/env python3

import rclpy
from rclpy.node import Node
from rclpy.qos import QoSHistoryPolicy, QoSProfile, QoSReliabilityPolicy
from ros2_plugin_proto.msg import RosMsgWrapper

from aimdk.protocol_pb2 import ProcessedAudioOutput, AudioVADState

import datetime
import os

class AudioSubscriber(Node):
    def __init__(self):
        super().__init__("audio_subscriber")

        # 音频缓冲区,按 stream_id 分别存储
        self.audio_buffers = {}  # {stream_id: bytearray()}
        self.recording_state = {}  # {stream_id: bool} 记录是否正在录音

        # 创建音频文件存储目录
        self.audio_output_dir = "audio_recordings"
        os.makedirs(self.audio_output_dir, exist_ok=True)

        qos_profile = QoSProfile(
            history=QoSHistoryPolicy.KEEP_LAST,
            depth=10,
            reliability=QoSReliabilityPolicy.BEST_EFFORT,
        )

        self.subscription = self.create_subscription(
            RosMsgWrapper,
            "/agent/process_audio_output/pb_3Aaimdk_2Eprotocol_2EProcessedAudioOutput",
            self.audio_callback,
            qos_profile,
        )

        self.get_logger().info("Started subscribing to noise-reduced audio data...")

    def audio_callback(self, msg):
        try:
            # Check if serialization type is pb
            if msg.serialization_type != "pb":
                self.get_logger().warn(f"Unsupported serialization type: {msg.serialization_type}")
                return

            # Convert the data field from list[bytes] to bytes
            audio_data_bytes = b"".join(msg.data)

            # Parse the message using the generated protobuf class
            processed_audio = ProcessedAudioOutput()
            processed_audio.ParseFromString(audio_data_bytes)

            self.get_logger().info(
                f"Received audio data: stream_id={processed_audio.stream_id}, "
                f"vad_state={processed_audio.vad_state}, "
                f"audio_size={len(processed_audio.audio_data)} bytes"
            )

            # Handle audio based on VAD state
            self.handle_vad_state(processed_audio)

        except Exception as e:
            self.get_logger().error(f"Error processing audio message: {e}")

    def handle_vad_state(self, processed_audio):
        """Handle different VAD states"""
        vad_state = processed_audio.vad_state
        stream_id = processed_audio.stream_id
        audio_data = processed_audio.audio_data

        # Initialize the buffer for this stream_id if it does not exist
        if stream_id not in self.audio_buffers:
            self.audio_buffers[stream_id] = bytearray()
            self.recording_state[stream_id] = False

        # VAD state name mapping
        vad_state_names = {
            AudioVADState.AUDIO_VAD_STATE_NONE: "No Voice",
            AudioVADState.AUDIO_VAD_STATE_BEGIN: "Voice Start",
            AudioVADState.AUDIO_VAD_STATE_PROCESSING: "Voice Processing",
            AudioVADState.AUDIO_VAD_STATE_END: "Voice End",
        }

        stream_names = {1: "Built-in Microphone", 2: "External Microphone"}

        self.get_logger().info(
            f"[{stream_names.get(stream_id, f'Unknown Stream {stream_id}')}] "
            f"VAD State: {vad_state_names.get(vad_state, f'Unknown State {vad_state}')} "
            f"Audio Data: {len(audio_data)} bytes"
        )

        # Handle audio data based on VAD state
        if vad_state == AudioVADState.AUDIO_VAD_STATE_BEGIN:
            self.get_logger().info("🎤 Voice start detected")
            # Start a new recording, clear the buffer
            self.audio_buffers[stream_id].clear()
            self.recording_state[stream_id] = True
            # Add current audio data
            if len(audio_data) > 0:
                self.audio_buffers[stream_id].extend(audio_data)

        elif vad_state == AudioVADState.AUDIO_VAD_STATE_PROCESSING:
            self.get_logger().info("🔄 Voice processing...")
            # If recording, continue adding audio data to the buffer
            if self.recording_state[stream_id] and len(audio_data) > 0:
                self.audio_buffers[stream_id].extend(audio_data)

        elif vad_state == AudioVADState.AUDIO_VAD_STATE_END:
            self.get_logger().info("✅ Voice end")
            # Add the last audio data
            if self.recording_state[stream_id] and len(audio_data) > 0:
                self.audio_buffers[stream_id].extend(audio_data)

            # Save the complete audio segment
            if (
                self.recording_state[stream_id]
                and len(self.audio_buffers[stream_id]) > 0
            ):
                self.save_audio_segment(bytes(self.audio_buffers[stream_id]), stream_id)

            # End recording
            self.recording_state[stream_id] = False

        elif vad_state == AudioVADState.AUDIO_VAD_STATE_NONE:
            # No voice state, do not record
            if self.recording_state[stream_id]:
                self.get_logger().info("⏹️ Recording state reset")
                self.recording_state[stream_id] = False

        # Output current buffer state
        if stream_id in self.audio_buffers:
            buffer_size = len(self.audio_buffers[stream_id])
            recording = self.recording_state[stream_id]
            self.get_logger().debug(
                f"[Stream {stream_id}] 缓冲区大小:{buffer_size} bytes, 录音状态:{recording}"
            )

    def save_audio_segment(self, audio_data, stream_id):
        """Save audio segment 16kHz, 16-bit, mono PCM"""
        if len(audio_data) > 0:
            timestamp = datetime.datetime.now().strftime("%Y%m%d_%H%M%S_%f")

            # Create subdirectory by stream_id
            stream_dir = os.path.join(self.audio_output_dir, f"stream_{stream_id}")
            os.makedirs(stream_dir, exist_ok=True)

            # Generate file name
            stream_names = {1: "internal_mic", 2: "external_mic"}
            stream_name = stream_names.get(stream_id, f"stream_{stream_id}")
            filename = f"{stream_name}_{timestamp}.pcm"
            filepath = os.path.join(stream_dir, filename)

            try:
                with open(filepath, "wb") as f:
                    f.write(audio_data)

                self.get_logger().info(
                    f"Audio segment saved: {filepath} (size: {len(audio_data)} bytes)"
                )

                # Record the duration of the audio file (assuming 16kHz, 16-bit, mono)
                sample_rate = 16000
                bits_per_sample = 16
                channels = 1
                bytes_per_sample = bits_per_sample // 8
                total_samples = len(audio_data) // (bytes_per_sample * channels)
                duration_seconds = total_samples / sample_rate

                self.get_logger().info(
                    f"Audio duration: {duration_seconds:.2f} seconds ({total_samples} samples)"
                )

            except Exception as e:
                self.get_logger().error(f"Failed to save audio file: {e}")

    def get_buffer_info(self):
        """Get information of all buffers (for debugging)"""
        info = {}
        for stream_id in self.audio_buffers:
            info[stream_id] = {
                "buffer_size": len(self.audio_buffers[stream_id]),
                "recording": self.recording_state[stream_id],
            }
        return info

def main(args=None):
    rclpy.init(args=args)

    audio_subscriber = AudioSubscriber()

    try:
        audio_subscriber.get_logger().info("Listening to noise-reduced audio data, press Ctrl+C to exit...")
        rclpy.spin(audio_subscriber)
    except KeyboardInterrupt:
        audio_subscriber.get_logger().info("Received exit signal, shutting down...")
    finally:
        audio_subscriber.destroy_node()
        rclpy.shutdown()

if __name__ == "__main__":
    main()

The program above depends on the Python package a2_aimdk and the ROS2 package ros2_plugin_proto. These packages are located in the prebuilt directory of the AimDK development kit. To install the Python package, use the following command: pip install prebuilt/a2_aimdk-3.0.0-py3-none-any.whl For the ROS2 package, you need to source the environment before use: source prebuilt/ros2_plugin_proto_aarch64/share/ros2_plugin_proto/local_setup.bash

Please note that the program above receives ROS2 messages, so the following environment variables need to be set:

bash
export ROS_DOMAIN_ID=232
export FASTRTPS_DEFAULT_PROFILES_FILE=/agibot/software/v0/entry/cfg/ros_dds_configuration.xml

The audio data consists of 16kHz, 16-bit, mono PCM (Little Endian). The output is clean human voice that has undergone noise reduction and echo cancellation, making it ready for direct use in ASR recognition.

The ProcessedAudioOutput message contains the following fields:

Field NameTypeDescription
headerHeaderStandard message header, containing the timestamp and message ID.
stream_iduint32Audio stream ID (1: Built-in Microphone, 2: External Microphone)
vad_stateAudioVADStateVoice Activity Detection (VAD) state.
audio_databytesNoise-reduced PCM audio data.

AudioVADState 枚举定义:

Enum valueValueDescription
AUDIO_VAD_STATE_NONE0No voice
AUDIO_VAD_STATE_BEGIN1Voice Start
AUDIO_VAD_STATE_PROCESSING2Voice Processing
AUDIO_VAD_STATE_END3Voice End

Note: In the current version, there is a known issue with the vad_state output on external microphones. Expected sequence: 122222222223 Actual sequence: 0111111111112 This issue only occurs with external microphones (built-in microphones work correctly) and is scheduled to be fixed in a future release. Workaround: We recommend manually applying a +1 offset to the state values in the current version.

8.2.3 Speaker Audio Playback

To use the speaker, you must first acquire the audio focus:

bash
ros2 service call /audio_5Fmsgs/srv/RequestAudioFocus audio_msgs/srv/RequestAudioFocus "{focus_requester: {pkg_name: audio_examples_sender, priority: 6, priority_weight: 1}}"

pkg_name refers to the ROS package name, which implies that the audio must ultimately be transmitted within the ROS package project structure. priority is fixed at 6, while priority_weight represents the audio weight and can be set to any integer between 1 and 100.

After acquiring the audio focus, you can play the PCM data stream returned by your TTS service using the audio streaming interface. The specific example for playing streaming audio through the speaker is provided below. Please note that this needs to be packaged as a ROS package.

bash
#!/usr/bin/env python3

import rclpy
from rclpy.node import Node
from builtin_interfaces.msg import Time as TimeMsg
from audio_msgs.msg import AudioPlayback,AudioInfo,AudioData  # 需确保 audio msg 可导入

class AudioPlaybackPublisher(Node):
    def __init__(self):
        super().__init__('audio_playback_publisher')
        self.pub = self.create_publisher(AudioPlayback, '/audiohal/audio/playback', 10)
        self.timer = self.create_timer(1.0, self.timer_callback)  # 每秒发送

    def timer_callback(self):
        msg = AudioPlayback()

        # 填写时间戳(字段名 attachments 中为 stamps)
        now = self.get_clock().now().to_msg()
        # 兼容性:优先使用 stamps,否则尝试 stamp
        if hasattr(msg, 'stamps'):
            msg.stamps = now
        elif hasattr(msg, 'stamp'):
            msg.stamp = now

        # 填写 info(AudioInfo)
        info = AudioInfo()
        info.channels = 1          # 单声道
        info.sample_rate = 16000   # 16kHz
        info.sample_format = 'S16LE'
        info.coding_format = 'pcm'
        msg.info = info

        # 填写 data(AudioData),示例给一小段原始 bytes
        sample_bytes = b'\x00\x01\x02\x03\x04\x05\x06\x07'  # 示例数据
        msg.data = AudioData(data=list(sample_bytes))

        msg.pkg_name = 'audio_examples_sender'
        msg.token_id = 'token-0001'

        self.pub.publish(msg)
        self.get_logger().info(
            f'Published AudioPlayback pkg={msg.pkg_name} token={msg.token_id} '
            f'data_len={len(msg.data.data)} sample_rate={msg.info.sample_rate}'
        )


def main(args=None):
    rclpy.init(args=args)
    node = AudioPlaybackPublisher()
    try:
        rclpy.spin(node)
    except KeyboardInterrupt:
        pass
    finally:
        node.destroy_node()
        rclpy.shutdown()


if __name__ == '__main__':
    main()

The program mentioned above is located in the agent directory of the AimDK Development Kit. It depends on the Python package a2_aimdk and the ROS2 package ros2_plugin_proto. These packages are located in the prebuilt directory of the AimDK Development Kit. Please follow the steps below to set them up: Python Package: Install it using pip: pip install prebuilt/a2_aimdk-3.0.0-py3-none-any.whl ROS2 Package: Source the environment before use: source prebuilt/audio_msgs_proto_aarch64/share/audio_msgs/local_setup.bash

8.3 Wake-up Result Reporting

If the agent is set to only_voice mode, Agibot also provides wake-up result reporting capabilities. The Topic interface is located at: /agent/wakeup/pb_3Aaimdk_2Eprotocol_2EWakeUpResult

The following example demonstrates how to retrieve wake-up result reports:

python
#!/usr/bin/env python3

import rclpy
from rclpy.node import Node
from rclpy.qos import (
    QoSDurabilityPolicy,
    QoSHistoryPolicy,
    QoSProfile,
    QoSReliabilityPolicy,
)
from ros2_plugin_proto.msg import RosMsgWrapper

from aimdk.protocol_pb2 import WakeUpResult


TOPIC = "/agent/wakeup/pb_3Aaimdk_2Eprotocol_2EWakeUpResult"


class WakeUpSubscriber(Node):
    def __init__(self):
        super().__init__("wakeup_result_subscriber")

        qos_profile = QoSProfile(
            history=QoSHistoryPolicy.KEEP_LAST,
            depth=10,
            reliability=QoSReliabilityPolicy.RELIABLE,
            durability=QoSDurabilityPolicy.VOLATILE,
        )

        self.subscription = self.create_subscription(
            RosMsgWrapper,
            TOPIC,
            self.wakeup_callback,
            qos_profile,
        )

        self.get_logger().info(f"Started subscribing to WakeUpResult: {TOPIC}")

    def wakeup_callback(self, msg):
        try:
            if msg.serialization_type != "pb":
                self.get_logger().warn(
                    f"Unsupported serialization type: {msg.serialization_type}"
                )
                return

            # Concatenate bytes
            raw_bytes = b"".join(msg.data)

            # Parse protobuf
            wakeup_result = WakeUpResult()
            wakeup_result.ParseFromString(raw_bytes)

            # Log output
            import json
            from google.protobuf.json_format import MessageToDict

            self.get_logger().info(
                f"WakeUpResult: {json.dumps(MessageToDict(wakeup_result, preserving_proto_field_name=True), ensure_ascii=False, indent=2)}"
            )

        except Exception as e:
            self.get_logger().error(f"Error parsing WakeUpResult data: {e}")


def main(args=None):
    rclpy.init(args=args)
    node = WakeUpSubscriber()

    try:
        node.get_logger().info("Listening to WakeUpResult, press Ctrl+C to exit...")
        rclpy.spin(node)
    except KeyboardInterrupt:
        node.get_logger().info("Exiting...")
    finally:
        node.destroy_node()
        rclpy.shutdown()


if __name__ == "__main__":
    main()

The program depends on the Python package a2_aimdk and the ROS2 package ros2_plugin_proto. These packages are located in the prebuilt directory of the AimDK Development Kit. Python Package: Install it using the following command: pip install prebuilt/a2_aimdk-3.0.0-py3-none-any.whl ROS2 Package: You need to source the environment before using it: source prebuilt/ros2_plugin_proto_aarch64/share/ros2_plugin_proto/local_setup.bash

Please note that since the program receives ROS2 messages, the following environment variables must be set:

bash
export ROS_DOMAIN_ID=232
export FASTRTPS_DEFAULT_PROFILES_FILE=/agibot/software/v0/entry/cfg/ros_dds_configuration.xml