Task-flow payload protocol (skill orchestration)

A task flow hands the robot a JSON payload string through startRequest(payload, ...) and taskRequest(payload, ...). This page is the reference for that JSON: its structure, field semantics, execution rules, and end-to-end SIMPLE / COMPLEX examples.

Difference from response.onSkill: onSkill dispatches a single skill in response to one ASR text from a passive callback (field semantics live in Semantic Skills). A task flow is active — it can orchestrate multiple actions in sequential / parallel / random order and even append new action groups on the fly, which makes it a better fit for "guide tasks", "point-of-interest navigation", or "multi-step action scripts".

Overall payload shape

⚠️ payload must be a JSON array — both startRequest and taskRequest take an array of action_group items. Even when you dispatch a single action_group, wrap it in an array ([{...}]); a bare object ({...}) is not accepted.

json

[
  {
    "type": "action_group",
    "execution_mode": "parallel",
    "continue_on_failure": false,
    "actions": [
      {
        "action_type": "tts",
        "config": { "tts": "hello" }
      }
    ]
  }
]

You can also stream multiple action_group items:

json

[
  { "type": "action_group", "actions": [ ... ] },
  { "type": "action_group", "actions": [ ... ] }
]

action_group fields

Field	Required	Default	Description
`type`	No	`action_group`	If present, must be `action_group`
`actions`	Yes	—	List of `action` items
`execution_mode`	No	`parallel`	`parallel` / `sequential` / `random`
`continue_on_failure`	No	`false`	Kept at group level for forward-compatibility, but the decisive flag is the `continue_on_failure` on each individual action

action fields

Each action has the shape:

json

{
  "action_type": "tts",
  "config": {},
  "continue_on_failure": false
}

action_type (required, string) — see "Supported action_types" below
config (required, object) — per-action_type configuration; see "action_type details" below
continue_on_failure (optional)
- Effective under sequential mode — when an action fails, setting this to true lets the remaining actions continue.
- Under parallel mode all actions have already been dispatched simultaneously; the failure status simply rolls up at group level.

Queueing semantics for multiple action_groups

This is the easiest pitfall for integrators.

If a single request carries multiple action_group items, the current implementation queues them and executes serially.
The corresponding *.execute_ack is sent only after every action_group in that request has finished.

So:

Whether actions inside a group run in parallel is controlled by execution_mode.
Multiple groups inside the same request do not run in parallel.

Supported action_types

Scenario	Recommended `action_type`	Supports completion wait
Motion	`motion`	Yes
TTS	`tts`	Yes
Expression	`emoticon`	No
Pilot task start / resume / stop	`pilot_task` (`fixed_line_intro` still accepted)	No
Active wake-up / quit duplex dialog	`setting`	No
Turn	`move`	Not recommended (unimplemented)

Explicit literals: motion / move / emoticon / tts / fixed_line_intro / setting.

action_type details

Motion (`motion`) — supports "fire-and-forget / wait-for-completion"

json

{
  "action_type": "motion",
  "config": { "id": 59 }
}

Common fields:

Field	Description
`id`	Single motion ID, or an array (random pick)
`motion_name`	Run the motion by name
`motion_type`	Motion type, common values `0`–`3`
`ani_path`	Run by animation path directly
`interrupt`	Whether to interrupt the previous motion
`dont_repeat`	When picking from an array, avoid repeats
`wait_only`	Only wait, do not dispatch a motion
`execution_scenarios.time`	Wait timeout (ms)
`execution_scenarios.wait_status` / `wait_statuses`	Wait status, default `kCompleted`

Fire-and-forget — omit execution_scenarios, returns SUCCESS immediately after dispatch:

json

{
  "action_type": "motion",
  "config": { "id": 59 }
}

Wait for completion:

json

{
  "action_type": "motion",
  "config": {
    "id": 59,
    "execution_scenarios": { "time": 5000 }
  }
}

Notes:

kFailed is also folded into the wait result.
On timeout the current implementation treats it as success.

TTS (`tts`) — supports "fire-and-forget / wait-for-completion"

json

{
  "action_type": "tts",
  "config": { "tts": "Hello, I'm starting the announcement" }
}

Common fields:

Field	Description
`tts`	Text to speak
`nlg`	NLG scene ID
`params`	NLG template parameters
`priority`	Priority
`interrupt`	Interrupt the current playback first
`interrupted`	Whether the new TTS can itself be interrupted
`execution_scenarios.time`	Wait timeout (ms)

Fire-and-forget:

json

{
  "action_type": "tts",
  "config": { "tts": "Do not wait for this one" }
}

Wait for completion:

json

{
  "action_type": "tts",
  "config": {
    "tts": "Wait until this finishes speaking",
    "execution_scenarios": { "time": 10000 }
  }
}

Notes:

TTS waits do not need wait_status.
The current implementation waits on kCompleted / kCanncelled / kFailed.
On timeout the current implementation treats it as success.

Expression (`emoticon`) — no completion wait

json

{
  "action_type": "emoticon",
  "config": { "id": 5 }
}

Common fields:

Field	Description
`id`	Expression ID, single value or array for random pick
`times`	Repeat count
`interval_ms`	Interval between repeats
`priority`	Expression priority
`video_path` / `video_path_list`	Play by video path
`mode`	Playback mode, `1/ONCE` or `2/LOOP`
`video_priority`	Video playback priority

Note: emoticon is fire-and-forget — you cannot wait for the expression to finish playing inside an action_group.

Pilot task (`pilot_task`) — no completion wait

Backed by the pilot_task plugin; used for guided-tour / point-of-interest task control.

Start a task:

json

{
  "action_type": "pilot_task",
  "config": {
    "control_type": "pilot_start_task",
    "cmd": "target_poi",
    "target_name": "{{target}}",
    "play_tts": true
  }
}

Resume:

json

{
  "action_type": "pilot_task",
  "config": {
    "control_type": "pilot_resume_task",
    "play_tts": true
  }
}

Stop:

json

{
  "action_type": "pilot_task",
  "config": {
    "control_type": "pilot_stop_task",
    "play_tts": true
  }
}

Common fields:

Field	Description
`control_type`	Required, operation kind
`title`	Required when starting a task
`play_tts`	Whether to play a result prompt

Notes:

pilot_task returns "command dispatched", not "task finished".
execution_scenarios cannot wait for the task to actually complete.
"Go to the next stop" uses fixed_line_intro_go_to_next_point; "return to start and finish" uses fixed_line_intro_go_to_init_point.

Settings (`setting`) — no completion wait

Active wake-up (start listening):

json

{
  "action_type": "setting",
  "config": { "active_wakeup": true }
}

Current behaviour:

Fires ProcessWakeUpResult({ .wakeup_type = kCustomTriggered })
Then explicitly calls StartDialog()
Used to open active listening / dialog
Only "trigger to start" is supported — you cannot wait for ASR first packet or VAD inside the action group.

Quit duplex dialog:

json

{
  "action_type": "setting",
  "config": { "quit_voice": true }
}

Used to close dialog listening / exit the dialog session.

Turn (`move`) — not recommended

turn_left / turn_right are currently treated as unimplemented; avoid them in streaming action_group.
If you only need translation, keep using move_left / move_right.
For real in-place turns, fill in the MoveDirection mapping first before exposing them.

End-to-end examples

SIMPLE mode: one-shot dispatch

Fits scenarios like "greet + emoticon + motion" where no follow-up is needed.

json

{
  "type": "agentsdk.flow.start.request",
  "flow_id": "flow_demo_001",
  "event_id": "event_flow_start_001",
  "mode": "simple",
  "payload": [
    {
      "type": "action_group",
      "execution_mode": "parallel",
      "continue_on_failure": false,
      "actions": [
        {
          "action_type": "tts",
          "config": { "tts": "Starting the simple flow" }
        },
        {
          "action_type": "emoticon",
          "config": { "id": 5 }
        },
        {
          "action_type": "motion",
          "config": {
            "id": 59,
            "execution_scenarios": {
              "time": 5000,
              "wait_status": "kCompleted"
            }
          }
        }
      ]
    }
  ]
}

In Java, construct TaskFlowRequest(agentSdk, agentId, flowId, FlowMode.SIMPLE) and pass the JSON above as the payload to startRequest.

COMPLEX mode: streaming append of action_groups

Fits multi-stage orchestration like "open active wake-up → start a guided tour → resume / cancel later".

① Open the flow (just `mode`, no actions yet)

json

{
  "type": "agentsdk.flow.start.request",
  "flow_id": "flow_demo_nav_001",
  "event_id": "event_flow_start_001",
  "mode": "complex"
}

② Append "active wake-up"

json

{
  "type": "agentsdk.flow.task.request",
  "flow_id": "flow_demo_nav_001",
  "event_id": "event_flow_task_001",
  "payload": [{
    "type": "action_group",
    "execution_mode": "sequential",
    "actions": [
      {
        "action_type": "setting",
        "config": { "active_wakeup": true }
      }
    ]
  }]
}

③ Append "start guided-tour task"

json

{
  "type": "agentsdk.flow.task.request",
  "flow_id": "flow_demo_nav_001",
  "event_id": "event_flow_task_002",
  "payload": [{
    "type": "action_group",
    "execution_mode": "sequential",
    "actions": [
      {
        "action_type": "tts",
        "config": { "tts": "Starting the guided tour" }
      },
      {
        "action_type": "fixed_line_intro",
        "config": {
          "control_type": "fixed_line_intro_start_task",
          "title": "Fixed-route tour",
          "play_tts": true
        }
      }
    ]
  }]
}

④ Resume

json

{
  "type": "agentsdk.flow.task.request",
  "flow_id": "flow_demo_nav_001",
  "event_id": "event_flow_task_003",
  "payload": [{
    "type": "action_group",
    "execution_mode": "sequential",
    "actions": [
      {
        "action_type": "fixed_line_intro",
        "config": {
          "control_type": "fixed_line_intro_resume_task",
          "play_tts": true
        }
      }
    ]
  }]
}

⑤ Cancel

json

{
  "type": "agentsdk.flow.task.request",
  "flow_id": "flow_demo_nav_001",
  "event_id": "event_flow_task_004",
  "payload": [{
    "type": "action_group",
    "execution_mode": "sequential",
    "actions": [
      {
        "action_type": "fixed_line_intro",
        "config": {
          "control_type": "fixed_line_intro_stop_task",
          "play_tts": true
        }
      }
    ]
  }]
}

For the Java-side SDK integration steps see Active Operations Guide - Task Flow and Active Examples - TaskFlowExample.