VEFX-Bench v1.0 — frozen on 2026-08-03

A comprehensive benchmark for evaluating video editing models across 300 videos, 9 task categories, and 3 quality dimensions.

This frozen protocol fixes the dataset, task taxonomy, evaluator checkpoint, and aggregation rule.

Download Dataset

Download the immutable benchmark snapshot from Hugging Face. It contains 300 source videos with their corresponding editing instructions and the v1.0 manifest.

Download frozen v1.0 snapshot

Dataset revision: 3bf997e7eb4fa0d0c2d56cce5ddfccc1dfbda235

Frozen Reproducibility Identifiers

Use these immutable identifiers to reproduce the v1.0 evaluation protocol exactly.

Hugging Face dataset revision: 3bf997e7eb4fa0d0c2d56cce5ddfccc1dfbda235
Manifest SHA-256: 277d89f5cf23af4163fe6ee120654210f51d19c6dee2d3aa76a3248471d1f333
Reward model revision: a15a8dbe1b3eb07ee0919e8de059f170436ec9ff
Checkpoint SHA-256: c3c0d03f770f0a73631206821922213de75413bb517f7d7a2fd9ab1f2c38f59d

Submission Format

After running your video editing model on all benchmark videos, package the results as follows:

Create a .zip file containing 300 edited videos.

Name files as 0000.mp4, 0001.mp4, …, 0299.mp4

Each video should be the edited version of the corresponding original video in the benchmark dataset.

Tip: Ensure all 300 files are present and correctly numbered. Missing, misnamed, or failed items receive the minimum score of 1.0 in IF, RQ, and EE before aggregation.

Task Categories

The benchmark covers 9 distinct editing categories:

Attribute Editing

Camera Angle Editing

Camera Motion Editing

Creative Edit

Instance Editing

Instance Motion Editing

Quantity Editing

Style Editing

Visual Effect Editing

Evaluation Metrics

Each video is evaluated along three complementary dimensions using the frozen VEFX-Reward model. Overall is the mean of item-level GeoAgg scores and is the primary ranking metric.

Exact aggregation order (N = 300)

i_j = (IF_j − 1) / 3, r_j = (RQ_j − 1) / 3, e_j = (EE_j − 1) / 3

g_j = 1 + 3 × (i_j² × r_j × e_j)^(1/4)

Overall = (1 / N) × Σ[j=1…N] g_j

The evaluator first computes GeoAgg for each item, then averages the 300 item-level values. It does not compute GeoAgg from the three dataset-level dimension means. The arithmetic Mean is diagnostic only and never determines rank.

GeoAgg ★

Overall (GeoAgg) ★

Mean of per-item weighted geometric aggregates; IF has weight 2. This is the primary ranking metric.

Range: 1-4 · Higher is better

Instructional Following (IF)

Measures how well the edited video follows the editing instruction.

Range: 1-4 · Higher is better

Render Quality (RQ)

Measures the visual rendering quality of the edited video.

Range: 1-4 · Higher is better

Edit Exclusivity (EE)

Measures whether only the intended region/attribute was edited without side effects.

Range: 1-4 · Higher is better

Scoring Rubric

Each dimension is scored on a 1–4 scale:

Score	Level	Description
4	Excellent	Fully satisfies the criterion with no noticeable issues.
3	Good	Mostly satisfies the criterion with minor shortcomings.
2	Fair	Partially satisfies the criterion with noticeable issues.
1	Poor	Fails to satisfy the criterion or has severe issues.

Submit Your Results