Welcome to this new module: AI Robotics Applications and LLM Integration in ROS2.

In this module, we are going to explore how to integrate Large Language Models into a real ROS2 robotic application.

The goal is not simply to “chat with a robot”. The goal is much more practical: we want to give the robot a set of well-defined skills, expose those skills through ROS2, and then use an LLM to reason about which skill to call, with which parameters, and in which sequence.

In other words, we are going to build a robotic application where the LLM does not directly control the robot joints. Instead, the LLM acts as a high-level reasoning layer that generates structured commands, usually in JSON format, and sends them to a ROS2 application layer.

The robot will execute only predefined and validated skills, such as:

move to a pose
pick an object
place an object
follow a toolpath
inspect a scene
execute a vision-based manipulation task

This is the correct way to think about LLMs in robotics: not as a replacement for motion planning, control, safety, or perception, but as an intelligent orchestration layer on top of a robust ROS2 architecture.

What We Are Building in This Module

Throughout this module, we will progressively build applications around the xArm6 robotic platform.

We will start from a clean simulation and control environment, then we will extend it step by step with custom ROS2 packages.

The reference repositories for this module are:

https://github.com/LearnRoboticsWROS/my_xarm6_simulation.git

and:

https://github.com/LearnRoboticsWROS/my_xarm6_app.git

The first repository, my_xarm6_simulation, will contain the simulation environment, robot description integration, MoveIt configuration, camera setup, and launch files.

The second repository, my_xarm6_app, will contain the application layer: custom ROS2 nodes, robot skills, trajectory generation, perception logic, and later the LLM interface.

This separation is important because it reflects a clean robotics software architecture:

Simulation / Robot Infrastructure
        ↓
Application Skills
        ↓
LLM Reasoning Layer

Why We Start with the Workspace

Before integrating AI, LLMs, perception, or advanced planning, we need a stable robotics foundation.

A Large Language Model is useful only if the underlying robot system is reliable.

If the robot description is broken, if MoveIt is not configured correctly, if the controllers are not running, or if the simulation is unstable, adding an LLM on top will only create confusion.

That is why this first lesson focuses on creating a clean ROS2 workspace and preparing the xArm6 environment correctly.

This will become the base for all the applications we will develop later.

What You Will Learn in This Lesson

By the end of this lesson, you will be able to:

Create a dedicated ROS2 workspace for AI robotics applications
Clone the xArm ROS2 stack
Clone the custom simulation and application repositories
Install dependencies with rosdep
Build the workspace with colcon
Source the environment
Launch and verify the robot setup

This may look like a simple setup lesson, but it is actually the foundation of the entire module.

Every advanced application we build later will depend on this workspace.

Step-by-Step Plan

In this lesson, we will:

1. Create a new ROS2 workspace called xarm_ws
2. Clone the xArm ROS2 repositories
3. Clone the custom simulation repository
4. Clone the custom application repository
5. Install missing dependencies using rosdep
6. Build the workspace with colcon
7. Source the workspace
8. Launch the robot and verify that the system works

The workspace will look conceptually like this:

xarm_ws/
└── src/
    ├── xarm_ros2/
    ├── my_xarm6_simulation/
    └── my_xarm6_app/

The exact package structure will evolve during the module, but the important concept is this: we keep the robot infrastructure, the simulation layer, and the application layer organized from the beginning.

Why xArm6?

We use the xArm6 because it is a good platform for learning modern robotic application development.

It gives us a realistic 6-axis robot structure, MoveIt integration, simulation support, and a good foundation for manipulation tasks.

During the module, we will use it to build examples such as:

blind pick-and-place
vision-based manipulation
simulated camera integration
CNC-style trajectory generation
robot skills exposed through ROS2
LLM-generated task plans
JSON-based command execution

The important point is that the concepts are not limited to xArm6.

The same architecture can be adapted to other industrial robots, as long as the robot exposes a proper ROS2 interface, MoveIt configuration, and control layer.

The Core Idea of This Module

The key architectural idea is simple:

The robot should not receive random natural language commands directly.
The robot should expose safe, predefined, testable skills.
The LLM should decide which skill to call and with which parameters.

For example, instead of asking the LLM to directly generate joint positions, we want the LLM to generate something like:

{
  "skill": "pick_object",
  "object": "red_cube",
  "target_bin": "left_tray"
}

Then our ROS2 application interprets this command, validates it, and executes the corresponding robot behavior.

This is how we keep the system structured, testable, and closer to how an industrial robotic application should be designed.

Lesson Outcome

At the end of this lesson, you will have a clean and working ROS2 workspace ready for the rest of the module.

You will not yet build the full LLM interface in this lesson.

Instead, we prepare the base environment that will allow us to build it properly later.

Once this setup is complete, we will be ready to move into the next lessons, where we start creating custom robot skills, motion commands, simulated manipulation tasks, and eventually the LLM integration layer.

The objective of this module is to show how AI can be integrated into robotics in a practical and realistic way: not as a magic black box, but as an intelligent layer built on top of a solid ROS2 robotic architecture.