A Study Design for AI Video Brand Consistency Across Outputs

Teams that use AI video generation for recurring content face one question before they scale a tool. Does the tenth video from the same brand guide still look and sound like the first? The pages that rank for this question assert consistency without a sample, a method or a date. This post sets out a study design any marketing team can run and publish.

The design applies when a team produces recurring video from the same brand guide and wants to compare tools, models or settings on whether outputs stay on brand from run to run. A study describes only the model versions, plans and settings used on the dates it ran. Vendors change models often, so results age and have to be rerun.

Why AI Video Outputs Drift From The Brand

AI video outputs drift because generative models rebuild every frame from a prompt and references, and Kling AI's own guide says that from a single photo the model must guess the unseen parts of a character. Tools that apply stored brand assets drift in different places, so a study has to score each layer of the brand separately.

A brand shows up in four layers, and each one fails in its own way:

• Identity assets: The logo, fonts and colours. These break when a logo is redrawn, a font is substituted or a colour shifts.
• Presenter or character: The face, clothing and build of a recurring person. These break when the face changes between shots or videos.
• Visual style: Lighting, colour grade, composition and camera feel. These break when one output looks like a different production.
• Voice and pronunciation: The voice profile and how product names are said. These break when a term is mispronounced or the voice changes.

The split between avatar video and generative video decides where drift appears. Template-led avatar tools place a scripted presenter on a layout and apply stored logos, fonts and colours. Generative tools render each scene from a prompt and reference inputs, so the presenter, the style and any on-screen logo are rebuilt on every run.

Academic benchmarks already define consistency at the frame level. VBench scores subject consistency by comparing the features of a subject across frames, and background consistency the same way for the scene behind it. That definition covers one clip. It says nothing about whether two clips made a week apart show the same brand.

Vendor claims point in different directions on how much input a model needs. Runway says Gen-4 generates consistent characters, objects and locations across scenes from a single reference image. Kling AI's guide to Subject Binding says that when only a single photo is available, the AI must guess the unseen parts of the character. A study treats both as claims to test, not as results.

How To Measure AI Video Brand Consistency

To measure AI video brand consistency, run the same standard briefs several times in each tool and have raters who cannot see the tool name score every output against the brand guide. Subjective scoring of video has its own standard in ITU-R BT.500, the ITU recommendation for subjective assessment of television image quality. The design below fits on one page.

• Define the unit as one output from one standard brief
• Define the arms as each tool, model and plan exactly as used
• Fix the inputs per brief, covering brief text, references, brand assets, aspect ratio and duration
• Run every brief several times in every arm
• Strip tool marks and randomise the order of outputs
• Have several raters score each output against the brand guide, layer by layer
• Measure rater agreement before comparing arms
• Record denominators, model versions, dates, cost and limitations

The fixed input is the brief. Write it once, in plain language, the way a human creative brief sets out what a video is for and what it must contain. Every arm receives the same words, the same reference files and the same brand assets. Anything that differs between arms becomes a variable the study has to record.

Repeat runs are what separate this design from a demo. One output per tool shows what a tool can do once. Several outputs per brief show how much a tool varies when nothing in the input changes, which is the question a brand team needs answered before scaling.

ITU-R BT.500 is the ITU standard for subjective video quality assessment, and the version in force is BT.500-15, approved in May 2023. This study's own design sets the scoring conditions. Raters see outputs under the same conditions, in a randomised order, and score against a fixed scale.

Practitioner guidance lands in the same place. The Cliprise evaluation scorecard recommends a fixed pack of four to six prompts with at least three runs per prompt for a first test, scores criteria on a 1 to 5 scale and applies pass or fail usability gates before any ranking.

Decision rule: A difference between two tools counts only when it is larger than the spread between repeat runs of the same tool. If two runs of one tool differ as much as the two tools do, the study has measured noise, not a tool difference.

Standard Briefs For AI Video Testing

A standard brief for AI video testing fixes the script, references, brand assets, duration and aspect ratio, and the Cliprise evaluation scorecard recommends a fixed pack of four to six prompts with at least three runs per prompt for a first test. The brief below lists every input to keep the same across tools.

Brief ID — {{brief-id}}
Brief type — {{brief-type}}
Brand — {{brand-name}}
Scene — {{scene}}
Presenter or character — {{presenter-or-character}}
Reference set — {{reference-set}}
Logo placement — {{logo-placement}}
Palette — {{palette}}
Font — {{font}}
Duration — {{duration}}
Aspect ratio — {{aspect-ratio}}
Script — {{script}}
Pronunciation terms — {{pronunciation-terms}}
Runs per tool — {{runs-per-tool}}

A small pack of brief types covers all four layers. Each type stresses a different part of the brand:

• Presenter explainer: A presenter delivers the script with a lower third and the logo. It tests the presenter, identity assets and pronunciation.
• Product scene: The product appears in a set scene with the palette and lighting from the brand guide. It tests visual style.
• Character in two locations: The same recurring character appears in two different settings. It tests whether the character holds when the scene changes.
• End card: The logo, a short line of text and the brand colours close the video. It tests identity assets and text rendering.

Word every brief identically for every tool. The habits that help anyone write prompts for AI video still apply, but a study cannot tune the wording per tool without turning prompt skill into a hidden variable.

Reference count differs between tools, so it has to be held constant. Kling AI's Subject Binding takes up to four reference images or 8-second video clips, and its Element Library needs at least two images per multi-image element. Runway describes Gen-4 consistency from a single reference image, and The Decoder reported in March 2025 that Gen-4 holds a character from one reference across different lighting, locations and treatments.

Reference rule: Give every tool the same number of references, or record the difference as a variable. A tool that receives four angles of a character and a tool that receives one photo are not running the same test.

Build the reference set once and reuse it. The same set of front, side and detail images that teams use to keep characters consistent in AI video serves as the fixed reference input for every arm of the study.

Blinded Visual Scoring And Rater Agreement

Blinded visual scoring only means something when raters agree, and under the Koo and Li guideline an intraclass correlation of 0.75 to 0.9 indicates good reliability while anything below 0.5 is poor. Blinding keeps raters from scoring the tool they expect to win, and the agreement check shows whether the scale itself works.

Set up scoring in five steps:

1. Strip identifying marks. Remove tool watermarks, interface marks and file names where the terms of use allow.
2. Randomise the order. Shuffle outputs across tools, briefs and runs so no rater sees one tool in a block.
3. Give raters the brand guide. Every score is a judgement against the guide, not against personal taste.
4. Score each layer separately. Use a 1 to 5 scale per layer with fixed anchors. 1 means off brand, 3 means recognisable with fixes, and 5 means publishable as is.
5. Use several raters. One rater's score is an opinion. Several raters make agreement measurable.

Academic precedent supports a panel of this size. EvalCrafter collected opinions from 7 users for each metric on a 1 to 5 scale, and then fitted its objective metrics to those opinions. The randomised order, shared brand guide and fixed scale above are this study's own design choices.

Measure agreement with the intraclass correlation coefficient (ICC). Koo and Li's 2016 guideline gives four bands:

• Below 0.5 is poor reliability
• 0.5 to below 0.75 is moderate
• 0.75 to below 0.9 is good
• 0.9 and above is excellent

Agreement rule: If rater agreement is below good, fix the scoring anchors before comparing tools. A tool difference measured by raters who do not agree with each other is not a finding.

Watermarks can break blinding. Videos made with Synthesia's free allowance carry a Synthesia watermark, and any rater who sees it knows the tool. Score outputs from paid plans, or crop every output the same way so that no arm looks different from the others.

House styles can also give a tool away. A rater who has used a tool for months may recognise its colour grade or motion. Record which raters have heavy experience with which tools, and check whether their scores differ from the rest of the panel.

Automated Consistency Metrics And Where They Stop

VBench, a benchmark suite of 16 dimensions for video generation models, scores subject consistency by comparing DINO features across the frames of one clip, which flags drift inside an output but not whether two outputs match the same brand guide. Automated metrics are a useful first screen, and human scoring stays the measure.

VBench scores background consistency with CLIP feature similarity across frames. It provides human preference annotations to validate its scores, and its authors report that its evaluations correlate highly with human preferences.

EvalCrafter uses 700 prompts and 17 objective metrics for text-to-video generation. For each metric, 7 users gave opinions on a 1 to 5 scale, and EvalCrafter fits coefficients so that its objective metrics line up with those opinions.

The two benchmarks show different things. VBench's subject and background consistency scores measure similarity across the frames of one clip, which catches a face that changes mid-shot or a background that warps. EvalCrafter shows how automated metrics can be fitted to a human panel. Neither benchmark is built to score whether an output matches a brand guide, or whether one output matches another made from the same brief.

Use automated metrics in two places:

• As a screen: Flag outputs with heavy frame-to-frame drift before human scoring, and record them as excluded with a reason.
• As a cross-check: Compare automated consistency within clips with human scores for the presenter or character layer, and report where they disagree.

Which Brand Controls Each Tool Offers

Avatar-led tools such as HeyGen and Synthesia hold identity through stored logos, fonts and colours, while generative tools hold characters through references, and since Adobe's multi-year partnership with Runway brought the Gen-4.5 model into Firefly, a study should record the model as well as the tool. The map below shows what each tool lets a study hold constant.

The five tools in scope cover both forms of AI video. Avatar-led tools are HeyGen and Synthesia. Generative tools are Runway, Adobe Firefly and Kling AI. A study compares avatar tools with avatar tools and generative tools with generative tools, so each comparison is like with like.

Runway

Layer controlled: Runway says Gen-4 keeps characters, objects and locations consistent across scenes from a single reference image. Reference inputs: One reference image combined with text instructions, which The Decoder reported in March 2025 holds a character across lighting, locations and treatments. Brand asset store: Brand Kits, with 1 on Pro and up to 3 on Max, and none listed on Standard. Where it applies: The Pro and Max plans. What it does not lock: A reference guides each generation but does not fix it, so the logo, text and small details are rebuilt on every run and need scoring per output. Cost meter: Credits per second of video. Standard's 625 monthly credits equal 52 seconds of Gen-4.5, Pro's 2250 credits equal 187 seconds and Max's 9500 credits equal 791 seconds.

Adobe Firefly

Layer controlled: Visual style, through over 20 controls for style, lighting, colour and composition, and refinement by describing changes. Reference inputs: Text prompts and refinements, with flexible resolutions and aspect ratios, plus partner models from Google, OpenAI, Runway and others. Brand asset store: Firefly Custom Models, trained on a brand's own images. Where it applies: Custom Models work through the Text to image feature, and training needs a trainer entitlement from the company's Adobe administrator. What it does not lock: Custom Models are a public beta for images, and Adobe's help page does not mention video, so they stay out of a video comparison. Cost meter: A video allowance per plan, with Firefly Pro allowing up to 40 five-second videos and Standard up to 20.

HeyGen

Layer controlled: Identity assets and the presenter, through the Brand System. Reference inputs: Primary and alternate logos, colours by picker or HEX, fonts for body, caption, display and heading text, reusable images and videos, and chosen avatars. Brand asset store: The Brand System, on the Creator, Pro, Business and Enterprise plans, with no limit on how many can be created. Where it applies: The Brand menu in AI Studio and the + icon in Video Agent. The Brand Glossary for pronunciation and translation is separate and must be selected each time a video or translation is created. What it does not lock: HeyGen Academy says the team can draw from the kit, and updates apply to future projects only, so older projects keep their old settings.

Synthesia

Layer controlled: Identity assets and the presenter, through brand kits. Reference inputs: Colour themes of 4 colours each, corner styles, up to two fonts, up to 6 avatars and 6 Spaces, and up to two logo versions. Brand asset store: Brand kits on Enterprise plans only. Other plans set colours, fonts and a logo by hand in a template. Where it applies: The brand menu in Syren, the video action menu in the editor and templates. What it does not lock: Brand Kit 2.0 applies the kit automatically when a new video starts, as a default, and the update describes no locking feature. Cost meter: Video minutes, with a free allowance of 10 minutes, then Starter adding up to about 12 minutes a month and Pro up to about 60 on top of that allowance, and Enterprise unlimited.

Kling AI

Layer controlled: The character. Kling says Subject Binding in Kling 3.0 is designed to eliminate character drift by anchoring facial structure, clothing texture and voice profile across shots. Reference inputs: Up to four reference images or 8-second clips, ideally front, side, back and detail. A video reference must be a 3 to 8 second clip of a single character. Brand asset store: The Element Library, which saves characters, items and scenes, with 2 to 4 images per multi-image element and voices bound from 5 to 30 seconds of audio. Where it applies: Image and video generation, called with @ in Omni, with up to 7 elements in Video 3.0 Omni and up to 3 bound elements in Video 3.0. What it does not lock: Kling's guide says that with only a single photo, the AI must guess the unseen parts of the character. Cost meter: Credits, with AI multi-view completion giving members 3 free uses a day and then 5 credits per use.

Model rule: Record the model as well as the tool. Runway's Gen-4.5 runs inside Firefly and on Runway's own platform, so a Runway versus Firefly comparison on that model is one model in two interfaces. For a tool pick rather than a method, see the best AI video tools for brand consistency.

How To Report AI Video Study Results

An AI video study result is comparable only when it reports its denominators (briefs, runs per brief, total and excluded outputs), the model versions and dates, and rater agreement as an intraclass correlation judged on the Koo and Li reliability bands. These fields are the bar a credible result clears.

Cost per usable output comes from the Cliprise scorecard, which defines it as total generation and processing cost divided by approved outputs. It turns a credit or minute meter into a number a budget holder can compare.

Academic benchmarks show the level of disclosure to aim for. EvalCrafter reports its 700 prompts, its 17 metrics and its 7 raters per metric. VBench reports its 16 dimensions and validates them against human preference annotations. A brand study reports the same kind of counts.

Spread rule: Report the run-to-run spread for each tool next to its mean. A mean without a spread hides the variance a brand team cares about most.

Many ranking articles on this topic quote brand-trust statistics with no sample, method or date. A result that carries the fields above can be checked, repeated and compared, which is what separates evidence from assertion.

Limitations Of AI Video Consistency Studies

An AI video consistency study describes only the model versions, plans and settings used during its window, so a result for Runway Gen-4.5 says nothing about the next Runway model even when it ships inside the same interface. Every study should state what it cannot show next to what it found.

The main limits to report:

• Model change: Adobe customers get early access to Runway's newest models inside Firefly, and Kling published its Subject Binding guide for Kling 3.0 in June 2026. Results age with each release.
• Budget: Credits and allowances cap how many runs a budget buys. Runway meters seconds of Gen-4.5 in credits, Firefly caps video generations per plan and Kling charges credits for multi-view completion.
• Blinding: Watermarks and recognisable house styles can tell raters which tool made an output.
• Rater pool: Raters from one team share the same habits and blind spots about the brand.
• Beta features: Firefly Custom Models are a public beta for images and stay out of scope until they are available for video.

Frequently Asked Questions