DROID: Everyday tasks · Franka FR3
Language-instructed tabletop tasks, each worded as a DROID annotation, on a Franka FR3 (DROID).
Cosmos3 Nano
Close the drawer
Success in 8 s
FLUX 3 Action
Pick up the screwdriver from the cup and put it on the table
Success in 10 s
MolmoAct2
Put the Rubik's cube in the storage box
Success in 11 s
G0.5
Move the mug to the left
Success in 14 s
GR00T
Pick up the cup and put it on the plate
Success in 9 s
| # | Model | Episodes | Reliability @ 240 s | Speed @ 240 s |
|---|---|---|---|---|
| % of task done | tasks/min | |||
| Loading the benchmark… | ||||
Reliability is the percent of the task done: the mean credit of how far each episode had got within the time limit. A task of several steps scores the steps done plus the credit of the current step, over the number of steps. Speed is tasks per minute: 60 over the mean time of the successes within the limit. The numbers cover the 278 of the 286 tasks that every model ran. Models share a rank when chance can explain the difference between their per-task Reliability. At 240 s the table shows the leaderboard's Reliability and rank: they come from the ranking model, fitted on all 286 tasks. ± is half the 95% range.
How reliable is it at each time limit?
The mean credit of how far the episodes had got within each time limit, with its 95% range. A success counts from its success mark, and a fail or a safety stop scores 0 from the end of its episode. An episode stopped early keeps how far it got. At 240 s the table shows the leaderboard's Reliability, from the ranking model, so there it can differ from this curve. Drag the marker to rank the table at another limit.
How far the episodes got
- Success
- At target
- Control
- Contact
- Reaching
- Floating
- Failed
- Safety stop
Each bar splits a model's episodes by how far they got: the verdict for a success, a fail or a safety stop, else the furthest stage marked in the current step. The number beside each stage is its credit for one step in Reliability.
More charts
When do the successes come?
The success marks of each model's episodes, smoothed, as a share of its episodes per second.
Is it faster, or more reliable?
Each line is a model at every time limit from 1 s to 240 s: its Speed in tasks/min across, its Reliability up. The dot is the current time limit, and the dashed lines are its 95% ranges. The axis ends at 6 tasks/min, so the quickest successes at a short limit fall beyond it.
Do two models really differ?
Episodes
Each episode and its own result, the newest first.