RoboJudge

Video generation models are promising foundations for embodied world models, but only if their generated rollouts are physically valid and faithful to task instructions. Multimodal large language models (MLLMs) are increasingly used to judge these properties, yet their agreement with human judgments on embodied videos has not been systematically evaluated. To address this gap, we introduce RoboJudge-Bench, which measures Physical Adherence and Instruction Alignment across 800 human-annotated videos from 8 source corpora and 10 video generators, spanning grippers, human hands, and dexterous hands. Existing general-purpose MLLMs show limited agreement with human judgments, while specialized video judges transfer poorly to embodied scenarios. We therefore construct RoboJudge-Train, comprising 12K videos with 25K human-verified scores and rationales, and train RoboJudge-9B with supervised learning followed by reinforcement learning. RoboJudge-9B reaches an overall Pearson correlation of 0.719 with human ratings, an absolute improvement of 0.089 over the strongest proprietary baseline in our evaluation, and also improves on VideoPhy2 and Physion-Eval.

RoboJudge-Train contains 12K embodied videos annotated with 25K human scores and rationales for PA and IA. The released training annotations contain 12,351 PA records and 11,520 IA records. RoboJudge-9B is the resulting embodied-specialized multimodal judge; model weights and video assets are hosted on Hugging Face.