8. Guide to using interaction secondary development
8. Guide to using interaction secondary development
For client-side secondary development related to interaction, two development modes are provided:
- Simple extension of A2 interaction
- Agibot retains its core interaction capabilities and provides fundamental interfaces for TTS, gestures, and facial expressions. These interfaces allow users to control the robot for basic workflows such as exhibition guiding and presentations, while the robot continues to fully support Agibot's standard voice interaction solution.
- Full takeover for secondary development
- Disable Agibot's interaction capabilities. Agibot provides the robot's microphone input (after onboard noise reduction), allowing the secondary development program to fully take over all voice interaction tasks.
The following table provides a detailed comparison of the two.
| Mode | Agibot Interaction Retained? | Interfaces Provided by Agibot | Internet Connectivity Requirements | Applicable Scenarios | Applicable Scenarios |
|---|---|---|---|---|---|
| Simple Extension of Agibot Interaction | Yes | TTS playback, Speaker playback | Both the Agibot interaction solution and TTS interfaces require an internet connection. | Developing simple exhibition guide demos with fixed workflow orchestration. | Relatively simple. Workload depends on workflow complexity but is generally low. |
| Full Takeover by Secondary Development | No | Noise-cancelled audio, Speaker | Internet is required for robot initialization, but not for subsequent usage. | Developing a custom, complete interaction solution. | High difficulty. Requires a complete R&D team; significant workload. |
8.1 Simple extension of A2 interaction
In this mode, interfaces provided by Agibot—such as expression control, action playback, TTS playback, and face recognition—can be used to implement specific functions. However, please note that most of these interfaces are not yet available. Detailed instructions will be provided once these interfaces are officially released.
8.2 Full control for secondary development
Interaction-based secondary development involves three key components: disabling Agibot's interaction capabilities, acquiring microphone audio, and utilizing the speaker. The following sections will detail how to proceed with secondary development across these three areas.
8.2.1 Disable the A2 interaction capability
To enable full takeover by secondary development, you must first disable Agibot's interaction capabilities so that only the microphone audio (including onboard noise reduction) is output. Please follow the steps below: Adjust the agent module to only_voice mode. You can use the SetAgentPropertiesRequest interface in the Microphone Management section to achieve this.
- only_voice: Outputs only the noise-cancelled microphone audio to /agent/process_audio_output. All downstream processing chains are disconnected.
The RPC calls required are as follows:bashcurl -i \ -H 'content-type:application/json' \ -X POST 'http://192.168.100.110:59301/rpc/aimdk.protocol.AgentControlService/SetAgentPropertiesRequest' \ -d '{ "contents": { "properties": { "2": "only_voice" } } }'
Please note that a robot restart is required for the changes to take effect. You can wait until you have completed the modifications in Step 2 and then restart the robot all at once. After the restart, you can verify the current interaction mode by using the GetAgentPropertiesRequest interface in the Microphone Management section to confirm if the change was successful. (To restore the default settings, simply change "only_voice" to "normal" in the SetAgentPropertiesRequest interface.))
8.2.2 Microphone Audio Acquisition
Note: To obtain the audio data described below, the robot must be connected to the internet for at least 2 minutes upon startup to complete audio-related authentication. Otherwise, no raw audio will be output. If you intend to use it offline, please ensure that this interface is outputting audio successfully before disconnecting from the network. After exiting the interaction loop, you can obtain the noise-reduced microphone audio from the robot via the /agent/process_audio_output topic. An example program for retrieving this audio is provided below:
#!/usr/bin/env python3
import rclpy
from rclpy.node import Node
from rclpy.qos import QoSHistoryPolicy, QoSProfile, QoSReliabilityPolicy
from ros2_plugin_proto.msg import RosMsgWrapper
from aimdk.protocol_pb2 import ProcessedAudioOutput, AudioVADState
import datetime
import os
class AudioSubscriber(Node):
def __init__(self):
super().__init__("audio_subscriber")
# 音频缓冲区,按 stream_id 分别存储
self.audio_buffers = {} # {stream_id: bytearray()}
self.recording_state = {} # {stream_id: bool} 记录是否正在录音
# 创建音频文件存储目录
self.audio_output_dir = "audio_recordings"
os.makedirs(self.audio_output_dir, exist_ok=True)
qos_profile = QoSProfile(
history=QoSHistoryPolicy.KEEP_LAST,
depth=10,
reliability=QoSReliabilityPolicy.BEST_EFFORT,
)
self.subscription = self.create_subscription(
RosMsgWrapper,
"/agent/process_audio_output/pb_3Aaimdk_2Eprotocol_2EProcessedAudioOutput",
self.audio_callback,
qos_profile,
)
self.get_logger().info("Started subscribing to noise-reduced audio data...")
def audio_callback(self, msg):
try:
# Check if serialization type is pb
if msg.serialization_type != "pb":
self.get_logger().warn(f"Unsupported serialization type: {msg.serialization_type}")
return
# Convert the data field from list[bytes] to bytes
audio_data_bytes = b"".join(msg.data)
# Parse the message using the generated protobuf class
processed_audio = ProcessedAudioOutput()
processed_audio.ParseFromString(audio_data_bytes)
self.get_logger().info(
f"Received audio data: stream_id={processed_audio.stream_id}, "
f"vad_state={processed_audio.vad_state}, "
f"audio_size={len(processed_audio.audio_data)} bytes"
)
# Handle audio based on VAD state
self.handle_vad_state(processed_audio)
except Exception as e:
self.get_logger().error(f"Error processing audio message: {e}")
def handle_vad_state(self, processed_audio):
"""Handle different VAD states"""
vad_state = processed_audio.vad_state
stream_id = processed_audio.stream_id
audio_data = processed_audio.audio_data
# Initialize the buffer for this stream_id if it does not exist
if stream_id not in self.audio_buffers:
self.audio_buffers[stream_id] = bytearray()
self.recording_state[stream_id] = False
# VAD state name mapping
vad_state_names = {
AudioVADState.AUDIO_VAD_STATE_NONE: "No Voice",
AudioVADState.AUDIO_VAD_STATE_BEGIN: "Voice Start",
AudioVADState.AUDIO_VAD_STATE_PROCESSING: "Voice Processing",
AudioVADState.AUDIO_VAD_STATE_END: "Voice End",
}
stream_names = {1: "Built-in Microphone", 2: "External Microphone"}
self.get_logger().info(
f"[{stream_names.get(stream_id, f'Unknown Stream {stream_id}')}] "
f"VAD State: {vad_state_names.get(vad_state, f'Unknown State {vad_state}')} "
f"Audio Data: {len(audio_data)} bytes"
)
# Handle audio data based on VAD state
if vad_state == AudioVADState.AUDIO_VAD_STATE_BEGIN:
self.get_logger().info("🎤 Voice start detected")
# Start a new recording, clear the buffer
self.audio_buffers[stream_id].clear()
self.recording_state[stream_id] = True
# Add current audio data
if len(audio_data) > 0:
self.audio_buffers[stream_id].extend(audio_data)
elif vad_state == AudioVADState.AUDIO_VAD_STATE_PROCESSING:
self.get_logger().info("🔄 Voice processing...")
# If recording, continue adding audio data to the buffer
if self.recording_state[stream_id] and len(audio_data) > 0:
self.audio_buffers[stream_id].extend(audio_data)
elif vad_state == AudioVADState.AUDIO_VAD_STATE_END:
self.get_logger().info("✅ Voice end")
# Add the last audio data
if self.recording_state[stream_id] and len(audio_data) > 0:
self.audio_buffers[stream_id].extend(audio_data)
# Save the complete audio segment
if (
self.recording_state[stream_id]
and len(self.audio_buffers[stream_id]) > 0
):
self.save_audio_segment(bytes(self.audio_buffers[stream_id]), stream_id)
# End recording
self.recording_state[stream_id] = False
elif vad_state == AudioVADState.AUDIO_VAD_STATE_NONE:
# No voice state, do not record
if self.recording_state[stream_id]:
self.get_logger().info("⏹️ Recording state reset")
self.recording_state[stream_id] = False
# Output current buffer state
if stream_id in self.audio_buffers:
buffer_size = len(self.audio_buffers[stream_id])
recording = self.recording_state[stream_id]
self.get_logger().debug(
f"[Stream {stream_id}] 缓冲区大小:{buffer_size} bytes, 录音状态:{recording}"
)
def save_audio_segment(self, audio_data, stream_id):
"""Save audio segment 16kHz, 16-bit, mono PCM"""
if len(audio_data) > 0:
timestamp = datetime.datetime.now().strftime("%Y%m%d_%H%M%S_%f")
# Create subdirectory by stream_id
stream_dir = os.path.join(self.audio_output_dir, f"stream_{stream_id}")
os.makedirs(stream_dir, exist_ok=True)
# Generate file name
stream_names = {1: "internal_mic", 2: "external_mic"}
stream_name = stream_names.get(stream_id, f"stream_{stream_id}")
filename = f"{stream_name}_{timestamp}.pcm"
filepath = os.path.join(stream_dir, filename)
try:
with open(filepath, "wb") as f:
f.write(audio_data)
self.get_logger().info(
f"Audio segment saved: {filepath} (size: {len(audio_data)} bytes)"
)
# Record the duration of the audio file (assuming 16kHz, 16-bit, mono)
sample_rate = 16000
bits_per_sample = 16
channels = 1
bytes_per_sample = bits_per_sample // 8
total_samples = len(audio_data) // (bytes_per_sample * channels)
duration_seconds = total_samples / sample_rate
self.get_logger().info(
f"Audio duration: {duration_seconds:.2f} seconds ({total_samples} samples)"
)
except Exception as e:
self.get_logger().error(f"Failed to save audio file: {e}")
def get_buffer_info(self):
"""Get information of all buffers (for debugging)"""
info = {}
for stream_id in self.audio_buffers:
info[stream_id] = {
"buffer_size": len(self.audio_buffers[stream_id]),
"recording": self.recording_state[stream_id],
}
return info
def main(args=None):
rclpy.init(args=args)
audio_subscriber = AudioSubscriber()
try:
audio_subscriber.get_logger().info("Listening to noise-reduced audio data, press Ctrl+C to exit...")
rclpy.spin(audio_subscriber)
except KeyboardInterrupt:
audio_subscriber.get_logger().info("Received exit signal, shutting down...")
finally:
audio_subscriber.destroy_node()
rclpy.shutdown()
if __name__ == "__main__":
main()
The program above depends on the Python package a2_aimdk and the ROS2 package ros2_plugin_proto. These packages are located in the prebuilt directory of the AimDK development kit. To install the Python package, use the following command: pip install prebuilt/a2_aimdk-3.0.0-py3-none-any.whl For the ROS2 package, you need to source the environment before use: source prebuilt/ros2_plugin_proto_aarch64/share/ros2_plugin_proto/local_setup.bash
Please note that the program above receives ROS2 messages, so the following environment variables need to be set:
export ROS_DOMAIN_ID=232
export FASTRTPS_DEFAULT_PROFILES_FILE=/agibot/software/v0/entry/cfg/ros_dds_configuration.xml
The audio data consists of 16kHz, 16-bit, mono PCM (Little Endian). The output is clean human voice that has undergone noise reduction and echo cancellation, making it ready for direct use in ASR recognition.
The ProcessedAudioOutput message contains the following fields:
| Field Name | Type | Description |
|---|---|---|
| header | Header | Standard message header, containing the timestamp and message ID. |
| stream_id | uint32 | Audio stream ID (1: Built-in Microphone, 2: External Microphone) |
| vad_state | AudioVADState | Voice Activity Detection (VAD) state. |
| audio_data | bytes | Noise-reduced PCM audio data. |
AudioVADState 枚举定义:
| Enum value | Value | Description |
|---|---|---|
| AUDIO_VAD_STATE_NONE | 0 | No voice |
| AUDIO_VAD_STATE_BEGIN | 1 | Voice Start |
| AUDIO_VAD_STATE_PROCESSING | 2 | Voice Processing |
| AUDIO_VAD_STATE_END | 3 | Voice End |
Note: In the current version, there is a known issue with the vad_state output on external microphones. Expected sequence: 122222222223 Actual sequence: 0111111111112 This issue only occurs with external microphones (built-in microphones work correctly) and is scheduled to be fixed in a future release. Workaround: We recommend manually applying a +1 offset to the state values in the current version.
8.2.3 Speaker Audio Playback
To use the speaker, you must first acquire the audio focus:
ros2 service call /audio_5Fmsgs/srv/RequestAudioFocus audio_msgs/srv/RequestAudioFocus "{focus_requester: {pkg_name: audio_examples_sender, priority: 6, priority_weight: 1}}"
pkg_name refers to the ROS package name, which implies that the audio must ultimately be transmitted within the ROS package project structure. priority is fixed at 6, while priority_weight represents the audio weight and can be set to any integer between 1 and 100.
After acquiring the audio focus, you can play the PCM data stream returned by your TTS service using the audio streaming interface. The specific example for playing streaming audio through the speaker is provided below. Please note that this needs to be packaged as a ROS package.
#!/usr/bin/env python3
import rclpy
from rclpy.node import Node
from builtin_interfaces.msg import Time as TimeMsg
from audio_msgs.msg import AudioPlayback,AudioInfo,AudioData # 需确保 audio msg 可导入
class AudioPlaybackPublisher(Node):
def __init__(self):
super().__init__('audio_playback_publisher')
self.pub = self.create_publisher(AudioPlayback, '/audiohal/audio/playback', 10)
self.timer = self.create_timer(1.0, self.timer_callback) # 每秒发送
def timer_callback(self):
msg = AudioPlayback()
# 填写时间戳(字段名 attachments 中为 stamps)
now = self.get_clock().now().to_msg()
# 兼容性:优先使用 stamps,否则尝试 stamp
if hasattr(msg, 'stamps'):
msg.stamps = now
elif hasattr(msg, 'stamp'):
msg.stamp = now
# 填写 info(AudioInfo)
info = AudioInfo()
info.channels = 1 # 单声道
info.sample_rate = 16000 # 16kHz
info.sample_format = 'S16LE'
info.coding_format = 'pcm'
msg.info = info
# 填写 data(AudioData),示例给一小段原始 bytes
sample_bytes = b'\x00\x01\x02\x03\x04\x05\x06\x07' # 示例数据
msg.data = AudioData(data=list(sample_bytes))
msg.pkg_name = 'audio_examples_sender'
msg.token_id = 'token-0001'
self.pub.publish(msg)
self.get_logger().info(
f'Published AudioPlayback pkg={msg.pkg_name} token={msg.token_id} '
f'data_len={len(msg.data.data)} sample_rate={msg.info.sample_rate}'
)
def main(args=None):
rclpy.init(args=args)
node = AudioPlaybackPublisher()
try:
rclpy.spin(node)
except KeyboardInterrupt:
pass
finally:
node.destroy_node()
rclpy.shutdown()
if __name__ == '__main__':
main()
The program mentioned above is located in the agent directory of the AimDK Development Kit. It depends on the Python package a2_aimdk and the ROS2 package ros2_plugin_proto. These packages are located in the prebuilt directory of the AimDK Development Kit. Please follow the steps below to set them up: Python Package: Install it using pip: pip install prebuilt/a2_aimdk-3.0.0-py3-none-any.whl ROS2 Package: Source the environment before use: source prebuilt/audio_msgs_proto_aarch64/share/audio_msgs/local_setup.bash
8.3 Wake-up Result Reporting
If the agent is set to only_voice mode, Agibot also provides wake-up result reporting capabilities. The Topic interface is located at: /agent/wakeup/pb_3Aaimdk_2Eprotocol_2EWakeUpResult
The following example demonstrates how to retrieve wake-up result reports:
#!/usr/bin/env python3
import rclpy
from rclpy.node import Node
from rclpy.qos import (
QoSDurabilityPolicy,
QoSHistoryPolicy,
QoSProfile,
QoSReliabilityPolicy,
)
from ros2_plugin_proto.msg import RosMsgWrapper
from aimdk.protocol_pb2 import WakeUpResult
TOPIC = "/agent/wakeup/pb_3Aaimdk_2Eprotocol_2EWakeUpResult"
class WakeUpSubscriber(Node):
def __init__(self):
super().__init__("wakeup_result_subscriber")
qos_profile = QoSProfile(
history=QoSHistoryPolicy.KEEP_LAST,
depth=10,
reliability=QoSReliabilityPolicy.RELIABLE,
durability=QoSDurabilityPolicy.VOLATILE,
)
self.subscription = self.create_subscription(
RosMsgWrapper,
TOPIC,
self.wakeup_callback,
qos_profile,
)
self.get_logger().info(f"Started subscribing to WakeUpResult: {TOPIC}")
def wakeup_callback(self, msg):
try:
if msg.serialization_type != "pb":
self.get_logger().warn(
f"Unsupported serialization type: {msg.serialization_type}"
)
return
# Concatenate bytes
raw_bytes = b"".join(msg.data)
# Parse protobuf
wakeup_result = WakeUpResult()
wakeup_result.ParseFromString(raw_bytes)
# Log output
import json
from google.protobuf.json_format import MessageToDict
self.get_logger().info(
f"WakeUpResult: {json.dumps(MessageToDict(wakeup_result, preserving_proto_field_name=True), ensure_ascii=False, indent=2)}"
)
except Exception as e:
self.get_logger().error(f"Error parsing WakeUpResult data: {e}")
def main(args=None):
rclpy.init(args=args)
node = WakeUpSubscriber()
try:
node.get_logger().info("Listening to WakeUpResult, press Ctrl+C to exit...")
rclpy.spin(node)
except KeyboardInterrupt:
node.get_logger().info("Exiting...")
finally:
node.destroy_node()
rclpy.shutdown()
if __name__ == "__main__":
main()
The program depends on the Python package a2_aimdk and the ROS2 package ros2_plugin_proto. These packages are located in the prebuilt directory of the AimDK Development Kit. Python Package: Install it using the following command: pip install prebuilt/a2_aimdk-3.0.0-py3-none-any.whl ROS2 Package: You need to source the environment before using it: source prebuilt/ros2_plugin_proto_aarch64/share/ros2_plugin_proto/local_setup.bash
Please note that since the program receives ROS2 messages, the following environment variables must be set:
export ROS_DOMAIN_ID=232
export FASTRTPS_DEFAULT_PROFILES_FILE=/agibot/software/v0/entry/cfg/ros_dds_configuration.xml