Code2Video-Compass: A Comprehensive Benchmark for General-Purpose Code-Based Video Generation across T2V, I2V, and V2V

Code2Video-Compass overview: three input modalities, a general-purpose code stack, 15 models, 10 video categories, and dual-level evaluation.
Overview of Code2Video-Compass. The benchmark evaluates 15 multimodal large language models (MLLMs) on code-based text-to-video (T2V), image-to-video (I2V), and video-to-video (V2V) generation. Models generate executable code from text and optional visual references using a general-purpose stack supporting HTML/CSS, SVG, Canvas, and Three.js, with Remotion rendering the resulting videos. It comprises 2,104 tasks spanning 10 categories and 24 subtypes, evaluated through a comprehensive dual-level framework combining 12 objective metrics with five subjective metrics.

Overview Video

Code2Video-Compass: Overview. This video introduces Code2Video-Compass, showcasing code-based video generation across T2V, I2V, and V2V settings, along with the benchmark's task design, evaluation framework, and representative model results.

Abstract

Dataset Statistics

Code2Video-Compass combines 385 independent tasks with 1,719 aligned tasks derived from 573 source videos, covering T2V, I2V, and V2V.

Data construction pipeline with independent and paired task sets.
Data construction pipeline. (A) Independent tasks are developed from curated text seeds, images, and video clips. (B) Paired tasks share a complete source video across T2V, I2V, and V2V. (C) Both sets undergo prompt construction, QA checklist development, and annotation, yielding 2,104 tasks.
Video category, motion, subject, event count, and duration distributions.
Statistics cover 958 records (573 paired sources + 385 independent tasks). Video categories, motion types, and subject types allow multiple labels.
Complexity distributions by category and prompt lengths for T2V, I2V, and V2V.
Category-wise construction, dynamic animation, and state-dependency complexity, and prompt lengths across input settings. Records with multiple categories contribute to each corresponding row.

Leaderboard

Results for 15 MLLMs, grouped into proprietary and open-weight models. The Overall tab reports pooled scores over all tasks across T2V, I2V, and V2V, as in the paper. Hover a column header for the metric definition.

Model results across T2V, I2V, and V2V.

Avg. Rank averages per-metric ranks across all 15 models, excluding Operational metrics (ES and TC); ties receive average ranks, and models are ordered by Avg. Rank within each group. The 12 objective metrics use the purple scale and the 5 subjective dimensions the green scale — darker is better. Column arrows mark each metric's direction (↑ higher is better, ↓ lower is better). Bold marks the best and underline the second-best score within each model group. ES is the delivered-video rate; TC the mean output tokens over tasks with recorded usage. A dash indicates a pending result.

Case Studies

Side-by-side model outputs with task-specific comparisons.

Citation

If you find Code2Video-Compass useful, please consider citing our paper:

Contact Us

If you have any questions about Code2Video-Compass, feel free to reach out to us at chenzh26@m.fudan.edu.cn.

We are actively updating this project, with more content coming soon.