Towards Reliable Human Evaluations in Gesture Generation: Insights from a Community-Driven State-of-the-Art Benchmark
CVPR, 2026
Rajmund Nagy, Hendric Voss, Thanh Hoang-Minh, Mihail Tsakov, Teodor Nikolov, Zeyi Zhang, Tenglong Ao, Sicheng Yang, Shaoli Huang, Yongkang Cheng, M Hamza Mughal, Rishabh Dabral, Kiran Chhatre, Christian Theobalt, Libin Liu, Stefan Kopp, Rachel McDonnell, Michael Neff, Taras Kucherenko, Youngwoo Yoon, Gustav Eje Henter. (2026). "Towards Reliable Human Evaluations in Gesture Generation: Insights from a Community-Driven State-of-the-Art Benchmark." CVPR, 2152-2164.
This work examines how human studies assess speech-driven 3D gesture generation and identifies inconsistent protocols and experimental design flaws that prevent reliable comparisons between methods. It proposes a standardised evaluation protocol for the BEAT2 motion-capture dataset and applies it in a large-scale crowdsourced benchmark of six recent models, each trained by its original authors. The benchmark assesses motion realism and speech-gesture alignment separately. The findings show that newer models do not always surpass earlier methods and that published claims of strong realism or alignment can fail under rigorous assessment. The study argues for disentangling motion quality from multimodal alignment to support accurate benchmarking. To enable reproducible studies and further evaluation research, the authors plan to release five hours of generated motion, more than 750 rendered study videos, an open-source rendering script, and 16,000 pairwise human preference votes.