Kubernetes Robot Operator for ROS/2 Based Robots

  • By robolaunch
  • Last update: Dec 16, 2022
  • Comments: 15

robolaunch Kubernetes Robot Operator

robolaunch Kubernetes Robot Operator manages lifecycle of ROS 2 based robots and enables defining, deploying and distributing robots declaratively.

kubectl-get-robots

kubectl-describe-robot

Table of Contents

Idea

The main idea of this project is to manage robots as Kubernetes custom resources. As a custom resource, a robot's lifecycle contains following operations and benefits.

  • Robot Lifecycle Management
    • Deployment
    • Update
    • Upgrade
    • Vertical Scaling
      • Adjusting robot's resources
  • Robot Observability
    • ROS observability tools (eg. rViz, Foxglove, ROS Tracker)
    • Exporting ROS nodes, topics, services, actions and bandwidth
  • GPU Acceleration
    • Simulation (Gazebo, Ignition)
    • VDI
  • Geoghraphic Distribution
    • Cloud-powered robot
    • Cloud-connected robot
  • Software development lifecycle
    • Cloud IDE
  • Connectivity
    • Robot-to-Robot Discovery
    • Node-to-Node Discovery

Refer robolaunch.io and project wiki for more architectural details and documentations.

Quick Start

Installation

Make sure your Kubernetes cluster meets prerequisites here.

Label one of your Kubernetes nodes.

kubectl label node <NODE> robolaunch.io/platform=true

Apply release YAML.

Deploy Your First Robot

You can try example robots under config/samples/. For example, to deploy Linorobot 2, apply the YAML below.

# linorobot2.yaml
apiVersion: robot.roboscale.io/v1alpha1
kind: Robot
metadata:
  name: linorobot2
spec:
  robot:
    nodeSelector:
      robolaunch.io/platform: "true"
    distro: foxy
    state: Launched
    tools:
      tracker:
        enabled: true
      cloudIDE:
        enabled: true
      bridge:
        enabled: false
      foxglove:
        enabled: false
      vdi:
        enabled: false
    mode: Single
    resources:
      storage: 15000
      cpuPerContainer: 800m
      memoryPerContainer: 512Mi
    namespacing: false
    workspaces:
    - name: linorobot-ws
      repositories:
      - name: master_br
        url: https://github.com/tunahanertekin/linorobot2
        branch: tuna
        launch:
          launchFilePath: linorobot2_gazebo/launch/gazebo.launch.py
          env:
          - name: LINOROBOT2_BASE
            value: 2wd
      build: Standard
kubectl apply -f linorobot2.yaml

After applying YAML, check robot's status by using:

watch "kubectl get robot linorobot2"

To see events and other robot-specific configurations, run:

kubectl describe robot linorobot2

Aims & Roadmap

  • Extending validation rules
  • Adding more ROS 2 robots
  • Supporting ROS GEMs
  • Extending simulation options
  • Comprehensive integration tests
  • Supporting older ROS distros
  • Supporting ROS topic encryption
  • Developing fleet interfaces for robot
  • Supporting storage classes that supports ReadWriteMany access mode

Contributing

Please see this guide if you want to contribute.

Download

robot-operator.zip

Comments(15)

  • 1

    Prepare ROS 2 Workspaces Over Kubernetes API

    What would you like to be added?

    Initial assumptions:

    • We have a robot.roboscale.io custom object which has it's volumes configured successfully
    • There are no running processes that use robot's volumes (other than the discovery server)

    Workspaces and repositories should be defined using robot's Kubernetes API. Workspaces should be able to include multiple Git-based repositories. Repositories should be cloned by URL and branch. Workspace configuration should be designed to be a one-time batch operation with a Kubernetes job.

    Why is this needed?

    Next ideal operation after creating a ROS 2 environment is to prepare ROS 2 workspace(s) to start running or developing a robot.

  • 2

    Populate Robot's Volumes

    What would you like to be added?

    This issue aims to populate robot's volumes by creating

    • persistent volume claims
      • /usr
      • /etc
      • /var
      • /opt
      • /tmp/.X11-unix
      • dynamic workspace path for robot (ROS 2 workspace)
    • configuration jobs
      • copier job - copies relevant directories under dynamically provisioned volumes
      • driver installer - installs GPU drivers if the cluster has Nvidia device plugins
      • source repository preparer - prepares source (ROS 2) repositories, updates or upgrades respectively

    Why is this needed?

    Robot images are determined at runtime based on cluster configuration. Preparing volumes for robot is a requirement to make robot assets persistent.

  • 3

    Create Clientset for Custom Resources

    What would you like to be added?

    Related to #32.

    It should be a clientset for API group robot.roboscale.io to manage robot-related resources externally.

    Why is this needed?

    It's needed to enable external management layers for robot, such as fleet manager.

  • 4

    PR: Manage Robot's Integrated IDE

    :herb: PR: Manage Robot's Integrated IDE

    Description

    robotides.robot.roboscale.io custom resource is created. It selects a robot and a VDI as targets(connection points).

    Closes #27

    Type of change

    • [x] New feature (non-breaking change which adds functionality)
    • [x] This change requires a documentation update

    How can it be tested?

    It can be tested by deploying a RobotIDE, specifying a Robot and a Robot VDI:

    apiVersion: robot.roboscale.io/v1alpha1
    kind: RobotIDE
    metadata:
      labels:
        robolaunch.io/target-robot: sample
        robolaunch.io/target-vdi: vdi-testing
      name: ide-testing
    spec:
      serviceType: NodePort
      ingress: false
      privileged: false
    
  • 5

    Configure Standalone Discovery Server (per robot)

    What would you like to be added?

    For CloudOnly robots, a standalone discovery server should be present which includes

    • a deployment/pod
    • a service

    Why is this needed?

    Seperate pods should be communicating over DDS/multicast to satisfy integrity. Otherwise a multicast-supporting CNI would be a key requirement. (which is still an exhaustive design for CloudConnected or CloudPowered robot cases)

  • 6

    PR: Managing Robot Artifact

    :herb: PR: Managing Robot Artifact

    Description

    robotartifacts.robot.roboscale.io is created. It carries robot's definition in field .template.

    Closes #42

    Type of change

    • [x] New feature (non-breaking change which adds functionality)
    • [x] This change requires a documentation update

    How can it be tested?

    It'll be tested in fleet operator.

  • 7

    Create Artifact Resource for Robot

    What would you like to be added?

    It's needed to manage a new resource called RobotArtifact that will be used as a placeholder for Robot.

    Information about CRD:

    • API Version: robot.roboscale.io/v1alpha1
    • Kind: RobotArtifact
    • RobotArtifact does not have a controller, it's only a placeholder resource for Robot.

    Why is this needed?

    It's needed to enable a marketplace for robots. Also it can be used as reference object when creating new Robot instance.

  • 8

    PR: Arrange Robot - Discovery Server Coupling

    :herb: PR: Arrange Robot - Discovery Server Coupling

    Description

    Cover these cases for discovery server configuration of robot:

    • Type: Server
    • Type: Client, Location: Same Cluster
    • Type: Client, Location: Another Cluster

    Closes #40

    Type of change

    • [x] New feature (non-breaking change which adds functionality)
    • [x] This change requires a documentation update

    How can it be tested?

    It can be tested by changing .spec.discoveryServerTemplate field of robot.

  • 9

    Make Discovery Server Loosely Coupled w/ Robot

    What would you like to be added?

    Robots should connect discovery servers by label selectors. Example: robolaunch.io/target-discovery-server

    Why is this needed?

    In fleet case, multiple robots should connect a discovery server and this tight coupling would be hard to manage in further cases.

  • 10

    PR: Deploying Robot w/ One YAML File

    :herb: PR: Deploying Robot w/ One YAML File

    Description

    .spec.buildManagerTemplate and .spec.launchManagerTemplates fields are added to Robot API.

    Closes #38

    Type of change

    • [x] New feature (non-breaking change which adds functionality)
    • [x] This change requires a documentation update

    How can it be tested?

    Example can be found under docs/examples/

  • 11

    Deploy BuildManager and LaunchManager from Robot's Manifest

    What would you like to be added?

    Currently, BuildManager and LaunchManager resources are being deployed as seperate resources from robot, only coupled by labels. These two resources should be deployed by Robot's manifest such as RobotDevSuite.

    • add .spec.buildManagerTemplate
    • add .spec.launchManagerTemplate

    Why is this needed?

    Deploying Robot with one YAML makes deployment processes more compact.

  • 12

    Manage a Custom Resource that Holds Resource Utilization of Robot's Components

    What would you like to be added?

    A controller should track

    • CPU usage
    • Memory usage
    • GPU usage
    • Network load

    periodically. Frequency for gathering information should be set in .spec. Controller can gather metrics from paths

    • /sys/fs/cgroup/cpu/cpuacct.usage
    • /sys/fs/cgroup/memory/memory.usage_in_bytes
    • /proc/net/dev etc.

    Why is this needed?

    Mentioned in #46.

  • 13

    Validate Custom Resource Fields with Regex

    What would you like to be added?

    Validating webhooks should check regex pattern of the field. Majority of these checks can be done using Kubebuilder markers.

    Why is this needed?

    It's needed to limit inputs from users in robolaunch Management Console.

  • 14

    Add ServiceExport Object for Discovery Server

    What would you like to be added?

    Discovery server custom resource currently maintains three Kubernetes resources:

    • a pod (if .spec.type is Server)
    • a service (if .spec.type is Server)
    • a config map

    Discovery server should maintain ServiceExport object to enable connection between two clusters over VPN.

    Why is this needed?

    It's needed to solve connectivity issues between two cluster's namespaces.

  • 15

    Make Robot's Launch Process Automated

    The Story

    Users should be able to declare how their robots are launched before deploying their robots. This feature gives the ability to run a robot automatically if the launch process is declared once.