RTBENCHv0.4

AvaturnBench-1 · Open Avatar Bench Initiative

The Generative Avatar Leaderboard

Pixel-level evaluation of real-time avatar models. Reproducible. Weekly.

Run 2026-05-02·Harness avaturnbench-1.0.0·Hardware: single A100 80GB·Commit a8f32c1
Try the comparison →Reproduce locally →View methodology →

Headline scoreboard

Sorted by Composite (0–1), descending. Click any column to re-sort.

1Avaturn-1
avaturn-1-0.1.0
open · MIT99.4%0.7103.42×4.830.990
2Anam
Mabyduck persona (default voice)
closed12.7%0.3201.21×1.420.298
3Tavus
phoenix-1 (CVI, us-east, tested 2026-04-30)
closed2.1%0.1801.08×0.310.150
4HeyGen
Interactive Avatar API (Streaming Avatar SDK, default avatar)
closed0.3%0.0601.01×0.140.081
5Synthesia
Express-2 real-time preview
closed0.0%0.0401.00×0.090.068
—Beyond Presence
Realtime Avatar API (pending capture)
pending————pending

Composite = mean of per-probe normalizations. IFC normalized as 1 − |v − 1|. Higher is better.

The four tests

All six probes →

GR

→

Generation Rate

Does the same audio twice produce the same video?

% of frames varying between two identical-input runs. A pre-rendered loop scores 0%.

primitive · Per-pixel L2 above codec floor

AVC

→

Audio→Motion Causality (off-mouth)

Does the face respond to audio outside the lipsync overlay?

Correlation between audio prosody and motion in the brows, gaze, and head pose under input perturbation.

primitive · RAFT × HuBERT, Pearson r

LWA

→

Listening-Window Activation

Does the avatar move while you speak — or only when it speaks?

Ratio of motion energy during the user-speaking window vs. the avatar-speaking window. Loops sit near 1.0.

primitive · Diarized optical-flow ratio

IFC

→

Idle-Loop Compression

How loop-like is the idle output compared to a real human?

A perfect loop is periodic and trivially compressible. We score against real human idle. 1.0 = indistinguishable from human; 0.0 = trivially compressible loop.

primitive · x264 CRF compression vs. real human idle

The proof

Same audio. Same vendor. Two runs. The frame hashes are computed by the harness on identical-input replay.

scenario · support-package-lookup

audio · sha256:6e1c4b9af…

✗ video hashes identical (loop)

Run 1 · video sha256:c4ad17b80…

Run 2 · video sha256:c4ad17b80…

A loop returns identical hashes; a generative model does not. The closed vendors above each return a single hash across N=20 identical-input replays. Avaturn-1 returns 20 distinct hashes.

Per-probe drill-down

All six AvaturnBench-1 probes. The open model is highlighted as the open one — not as a winner; the chart speaks.

Generation Rate

% of frames varying between two identical-input runs. A pre-rendered loop scores 0%.

primitive · Per-pixel L2 above codec floor

GR · all vendors · run 2026-05-02

open closed pending capture

Run history

Every weekly run is a frozen JSON snapshot. PRs to data/runs/ add the next entry.

Reproduce

Single Docker command. Single A100. Wall-clock ≈ 12 minutes per vendor.

docker run --gpus all --rm \
  ghcr.io/avaturn/avaturnbench-1:1.0.0 \
  bench all \
  --scenarios scenarios-v1 \
  --replays 20 \
  --out ./run-2026-05-02.json
harness
avaturnbench-1.0.0
commit
a8f32c19d2c7
hardware
single A100 80GB
scenario pack
scenarios-v1 (5 emotion classes, 90s each, N=20 identical-input replays)
Bench source →This site's source →

Submit a model

Three steps. Maintainers re-run on the official hardware before merging — this is what makes the comparison apples-to-apples.

  1. step 1

    Implement BenchAdapter

    Subclass BenchAdapter in your model repo and implement stream(audio_chunk) → frames. Reference adapters live in avaturnbench-1/adapters.

    git clone https://github.com/avaturn/avaturnbench-1
cd avaturnbench-1/adapters
cp -r reference my-model && $EDITOR my-model/adapter.py

  2. step 2

    Run the bench locally

    Single docker command. Outputs a JSON file matching data/runs/<date>.json.

    docker run --gpus all --rm \
  -v $PWD/my-model:/adapter \
  ghcr.io/avaturn/avaturnbench-1:1.0.0 \
  bench my-model --out my-model.json

  3. step 3

    Open a PR

    Add your vendor entry to data/vendors.json and your run fragment to the next weekly snapshot. Maintainers re-run on the official A100 before merging.

    git checkout -b add-my-model
cp my-model.json data/runs/<next-run-date>.json
gh pr create -t "Add my-model" -b "..."
Open a PR →

Methodology, in one paragraph

AvaturnBench-1 measures whether an avatar system computes per-frame from input audio, or plays back a pre-rendered cycle with a phoneme-aligned mouth overlay. Six probes operate on raw video and input audio. Primitives are off-the-shelf: CLIP-L, RAFT, HuBERT, FER, and x264 compression. There are no human raters. The harness ships as a single Docker image.

Read the full methodology →
← Back to arena

Methodology — AvaturnBench-1

AvaturnBench-1 is a six-probe pixel-level evaluation harness for real-time avatar systems. Every probe operates on raw video and input audio. There are no human raters. The CV/ML primitives are off-the-shelf: CLIP-L, RAFT, HuBERT, FER, and x264 compression.

Why these six

GR, XPD, and IFC are definitional loop-catchers. A pre-rendered animation cycle cannot pass them without rebuilding the architecture. They are not gameable with a prompt or a fine-tune; they require generative inference. AVC is the lipsync-overlay killer: it measures the face outside the mouth region, where closed systems have nothing to perturb. LWA is the listening test. EVD is the screenshot test in numbers.

Scenario pack

Five emotion classes (joke, grief, neutral, anger, surprise). 90 seconds per scenario. N=20 identical-input replays per scenario. Audio is fixed and bit-identical across replays per the harness recording layer.

Probes

GR · Generation Rate

primitive · Per-pixel L2 above codec floor

Does the same audio twice produce the same video?

% of frames varying between two identical-input runs. A pre-rendered loop scores 0%.

GR  = mean_t [ ‖frame_a(t) − frame_b(t)‖_2 > 2·codec_floor ]   over identical-input runs a, b

rationale · A pre-rendered cycle returns identical frames on identical input. GR is the definitional loop-catcher.

XPD · Cross-Run Pixel Diversity

primitive · CLIP-L embedding distance

How much does output vary across N identical-input runs?

Spread of frame-embeddings across N=20 identical-input runs.

XPD = mean_{i,j} ‖CLIP-L(frame_i) − CLIP-L(frame_j)‖_2        over N=20 identical-input runs

rationale · Embedding spread across N runs distinguishes architectures that re-roll the rendering pass from those that re-play it.

AVC · Audio→Motion Causality (off-mouth)

primitive · RAFT × HuBERT, Pearson r

Does the face respond to audio outside the lipsync overlay?

Correlation between audio prosody and motion in the brows, gaze, and head pose under input perturbation.

AVC = pearson_r( HuBERT(audio), RAFT(off-mouth optical flow) )

rationale · Closed systems pass lipsync evals because they paint phoneme-aligned mouth shapes. AVC measures the rest of the face.

LWA · Listening-Window Activation

primitive · Diarized optical-flow ratio

Does the avatar move while you speak — or only when it speaks?

Ratio of motion energy during the user-speaking window vs. the avatar-speaking window. Loops sit near 1.0.

LWA = mean(motion_energy | user_speaking) / mean(motion_energy | avatar_speaking)

rationale · A loop plays its idle cycle while the user speaks. Listening avatars react during the user-speaking window.

EVD · Emotional-Valence Differentiation

primitive · FER + Fisher's discriminant ratio

Different face on a joke vs. on grief — or the same face every time?

Class separability of facial-expression embeddings across 5 emotion classes.

EVD = Fisher_ratio( FER_embeddings ; { joke, grief, neutral, anger, surprise } )

rationale · Same loop, same expression vector, low Fisher ratio. The ‘polite smile on grief’ failure mode formalized.

IFC · Idle-Loop Compression

primitive · x264 CRF compression vs. real human idle

How loop-like is the idle output compared to a real human?

A perfect loop is periodic and trivially compressible. We score against real human idle. 1.0 = indistinguishable from human; 0.0 = trivially compressible loop.

IFC = 1 − | x264_CRF(avatar_idle) − x264_CRF(human_idle) | / x264_CRF(human_idle)

rationale · A perfect loop is periodic and trivially compressible. We score against real human idle.

How this differs from the Anam/Mabyduck eval

Anam / Mabyduck (Oct 2025)AvaturnBench-1 (May 2026)
5-point Likert scalePixel- and audio-level CV/ML primitives
178 paid raters0 raters
2-minute ‘20 Questions’ — sterile, low-stakes90s open scenarios; 5 emotion classes; identical-input replay
Loop detection: not measuredLoop detection: primary axis (GR, XPD, IFC)
Self-reported satisfactionObserved pixel evidence
Commissioned by AnamOpen source, MIT, PR-driven
No public harnessOne docker command

Composite

Composite is the unweighted mean of per-probe normalizations to [0,1]. Five of the six probes are higher-is-better and divided by the field max. IFC is normalized as 1 − |v − 1| so that values close to real human idle compression score 1, and trivially-compressible loops score 0. We chose unweighted mean because per-probe weighting is the kind of knob a consortium negotiates for v2; v1 is deliberately blunt.

Disclosure

Closed-vendor scores reflect tests against published public APIs on the dates listed per vendor row. Each vendor page lists the exact endpoint and capture date. Vendors are welcome to PR a dated correction with their own rerun command line.