Physical AI Leaderboard
A demo does not show if a physical-AI model is ready for production. PhAIL measures this on real robots, with real commercial tasks, and with the metrics that an operations team uses.
Where this is going
Today, PhAIL is a leaderboard with two benchmarks. The larger goal is that PhAIL does the real-robot evaluation of VLA (vision-language-action) models, and you do the research. A lab can publish its results on the leaderboard, or keep them private. The harness that we run will become a toolkit that you can run yourself.
Read the paper
Our paper on arXiv gives the PhAIL method: what PhAIL measures, and why.
Benchmarks
Each benchmark has a page with its ranking and its charts.
- DROID: Everyday tasks. Language-instructed tabletop tasks, each worded as a DROID annotation, on a Franka FR3 (DROID).
- Pick and Place, v1.0. Bin-to-bin pick and place of four objects on a Franka FR3 with a Robotiq 2F-85 gripper (DROID).
DROID: Everyday tasks has 101 tasks. An operator watches each episode and marks each stage that the robot reaches. At the end, the operator gives a verdict: Success, Fail, Ran out of time, or Safety. The console stops an episode at 240 s. An annotator then reviews the recording. Where the annotator labelled an episode, the annotator's marks decide.
In Pick and Place, the robot moves a known number of items from one tote to a second tote. Each run uses one of four objects. The v1.0 release page gives the full protocol.
Metrics
Each benchmark gives a model two numbers, Reliability and Speed. The leaderboard ranks the models by Reliability. Both numbers count only what the model does within a time limit. On a benchmark page, you can move the time limit.
DROID: Everyday tasks. The leaderboard shows both numbers at 240 s.
Reliability – Percent of the task done, from the ranking model: each model gets a skill and each task a difficulty, fitted to how far every episode got within 240 s. Reliability is the expected credit over the tasks: 1 for a success, 0.4 at the target, 0.3 in control, 0.1 at contact, else 0.
Speed – Tasks per minute while the model succeeds: 60 over the mean time to the success mark, over the successes within 240 s.
Pick and Place. The leaderboard shows both numbers at 120 s.
Reliability – Percent of items placed, each within 120 s of the previous item.
Speed – Items per minute while the model places them: 60 over the mean time of the items placed within 120 s.
On Pick and Place, each of the four objects has the same weight.
On DROID: Everyday tasks, the leaderboard's Reliability comes from a ranking model. The model works like the scoring of an exam: each model gets a skill, and each task gets a difficulty. The chance that an episode gets at least as far as a stage rises with the skill minus the difficulty, and each model has its own spacing of the stages. The fit picks the skills and difficulties that make the recorded episodes most likely. A model's Reliability is then its expected credit over the tasks. The episode lists name each task's difficulty by how much of the task a model of average skill gets done: Easy 60% or more, Medium 40 to 60%, Hard 20 to 40%, Very hard under 20%. An episode that the operator stopped early keeps the furthest stage that it reached. The charts on the benchmark page show the episodes as recorded, over the tasks that every model ran.
Each number shows ± half of its 95% range. The range comes from 1,000 bootstrap draws, and each draw samples the data again, with replacement. On Pick and Place, a draw samples the runs of each object. On DROID: Everyday tasks, a draw samples the tasks and fits the ranking model again, and all the models use the same draw.
Two models share a rank when the data cannot separate them. On Pick and Place, a logrank test compares the item times of two models, and it must give p below 0.05 after a Bonferroni correction: p times the number of model pairs. On DROID: Everyday tasks, a model ranks above another when it has the higher Reliability in at least 95% of the 1,000 draws. A shared rank shows as a range, for example 2–3.
Getting started
To see the v1.0 fine-tuning dataset, run one command:
uv run --with positronic \
python -m positronic.cfg.phail.v1_0pip install positronic
python -m positronic.cfg.phail.v1_0- The leaderboard shows the scores of each model on each benchmark. On each benchmark page, you can see the ranking and the charts at a different time limit.
- The v1.0 release page gives the model weights, the recorded evaluation runs, the protocol, the hardware specification, and the steps to reproduce the results.
- To get an evaluation, read the options on the eval page, or send an email to [email protected]. The results can go on the leaderboard, or stay private to your lab. For v1.0, we do the onboarding by hand.
Pick and Place release: v1.0
The task is bin-to-bin order picking of four object classes: towels, wooden spoons, scissors, and batteries. The hardware is one Franka Research 3 arm with a Robotiq 2F-85 gripper and two Stereolabs ZED cameras. The release evaluates four models: OpenPI π0.5, NVIDIA GR00T N1.6, Hugging Face SmolVLA, and Action Chunking Transformer.
The v1.0 release page gives the full hardware specification, the protocol, the dataset, the model weights, and the steps to reproduce the results. The v1.0 white paper gives the evaluation method of the release.
Open consortium
An open consortium governs PhAIL, so no single company controls what PhAIL measures or how. Nebius is the founding compute partner, and Toloka is the data partner. Partners and how to join →
Contact
Send questions, feedback, or a model for evaluation to [email protected].