Frontier data research for physical agents

Flowstone makes training data for frontier model labs building physical agents.

One episode, six layers

A robot picks up a mug and sets it on a shelf.

Select a layer to see what it records and what it gives a model.

Layer	What it records	What it gives a model
Video	What the agent sees, from head and wrist cameras.	The scene, the objects, and how the task unfolds.
Depth	The 3D position of visible surfaces.	The geometry for reaching and placing accurately.
Pose	Joint angles and hand or body position over time.	A record of how the body moved through the task.
Action	The commands sent to the motors and gripper.	The output a policy learns to produce.
Force	Force and torque at the wrist during contact.	How hard to grip and push, which video shows poorly.
Touch	Pressure and slip at the fingertips.	Contact and slip at the fingertips, as they happen.

Some data moves the curve

Adding data does not always make a model better. Flowstone makes the data that does.

Model performance Data

What makes the data useful

Transfer

Sensors and tasks chosen so the data carries over to a robot it was not recorded on.

Structure

Contact, force, and object state aligned and represented so that models can use them.

Realism

Real interaction where simulated or generated data cannot stand in.

Measurement

Episodes kept for how much they improve a physical agent on real hardware.

Data cards

What each dataset contains, the format it comes in, and how to get access.

Head camera view of one gripper holding a slice of bread beside a toaster.

Left wrist camera view of the toaster on a wooden table.

Right wrist camera view of the bread held between the fingers, beside a holder of slices.

Simulation · dual-arm

SFT Data in Simulation

Simulated demonstrations as multimodal agent conversations. Each turn gives the robot its state and three camera images, then a short first-person chain of thought and one tool call.

OpenAI chat and tool call format, three cameras per step

Episodes: 3,500
Tasks: 35
Steps: 89,056
Images: 277,668

Read the data card

Head camera view of both arms placing a gold star on a small Christmas tree.

Left wrist camera view of the gold star, close up.

Right wrist camera view of the other arm and the top of the tree.

Real robots · dual-arm

SFT Data on Real Robots

Household and tabletop tasks in the same conversation format. Some episodes include an attempt that does not work: the chain of thought says so once the views show it, and the next steps try again.

OpenAI chat and tool call format, three cameras per step

Episodes: 500
Tasks: 175
Steps: 75,375
Images: 227,625

Read the data card

Contact

For access to the data.

ethan@flowstonedata.com