Data Pipeline

How Artemis ingests, preprocesses, stores, and manages your training datasets.

Supported Data Formats

Artemis accepts robotics data in a wide range of standard formats. Files are automatically detected and parsed on upload:

Format	Extension	Use Case
ROS Bag	.bag, .db3	Full ROS 2 topic recordings — preferred format.
CSV Telemetry	.csv	Tabular sensor readings, joint states, or IMU data.
Image Archive	.zip, .tar.gz	Labelled image sets for object detection tasks.
LIDAR Cloud	.pcd, .bin	Point cloud data from 3D LIDAR sensors.
JSON Telemetry	.json, .jsonl	Arbitrary structured sensor data.
HDF5	.h5, .hdf5	Large multi-modal datasets with metadata.

Uploading Data

Upload a dataset with the CLI:

bash

kairo data upload ./recording.bag \
  --robot rob_a1b2c3d4 \
  --label "arm-training-run-3" \
  --tags "production,v2"

For large datasets, uploads are chunked and resumable:

bash

# If an upload is interrupted, resume it
kairo data resume --upload-id upl_xyz789

ℹ

Datasets larger than 500 MB are automatically split into 100 MB chunks and uploaded in parallel. Progress is visible in the dashboard under Data → Uploads.

Automatic Preprocessing

After upload, Artemis runs an automatic preprocessing pipeline before training:

Validation — Checks file integrity, detects corrupt frames, and validates message schemas.
Normalisation — Scales sensor values to a consistent range per topic type.
Resampling — Aligns multi-topic recordings to a unified timestamp grid.
Augmentation — Applies optional data augmentation (noise injection, frame flipping) to improve model generalisation.
Train/Val split — Automatically splits data into 80% training and 20% validation sets.

Storage Limits

Plan	Storage	Max File Size	Retention
Starter	1 GB	500 MB	90 days
Builder	50 GB	10 GB	1 year
Pro	500 GB	100 GB	Unlimited

⚠

When you reach 80% of your storage quota, Artemis sends an email alert. At 100%, new uploads are paused — existing data and active deployments are unaffected.

Managing Datasets

bash

# List all datasets
kairo data list

# Show details and preprocessing status
kairo data info --dataset ds_xyz123

# Delete a dataset (irreversible)
kairo data delete --dataset ds_xyz123

✦

Deleting a dataset does not delete any models that were trained from it. Model weights are stored separately and are unaffected by dataset deletion.

←

AI Models

ROS 2 Guide

→

Was this page helpful?