Computer Vision · Color Science · Image Analytics

Cinema Color
Intelligence

Extracting palette fingerprints from film frames and surfacing measurable aesthetic signatures by director, genre, and narrative arc.

Color grading is one of the least-quantified craft elements in filmmaking. Cinematographers and directors spend enormous effort building distinctive visual languages, but evaluating those decisions has always been subjective. This pipeline makes it empirical: extract every frame, convert to a perceptually uniform color space, cluster dominant palettes, and aggregate into per-film and per-director fingerprints that can actually be compared.

The result is a dataset of color behavior across films, with enough structure to ask questions like: does this director consistently shift toward cooler tones during third-act tension? Does warm palette density predict critic reception in a given genre? Where in a film does color entropy peak?

The Setup
Concept 01
LAB Color Space
RGB values are device-dependent and perceptually non-uniform. Converting to CIELAB (L*, a*, b*) separates luminance from chroma and makes Euclidean distance meaningful. A ΔE of 1.0 is approximately the threshold of human perception.
Concept 02
K-Means Palette Clustering
Each frame is downsampled and its pixels clustered into k=5 dominant colors in LAB space. The cluster centroids are the frame's palette fingerprint. Five clusters balances descriptive richness against noise from incidental background pixels.
Concept 03
Warm/Cool Ratio
A proxy for emotional temperature. Computed by binning each dominant color's hue angle: reds, oranges, and yellows (0-60° and 300-360°) count as warm; cyans and blues (180-270°) as cool; remaining hues as neutral. Aggregated as a ratio per scene and per film.
Concept 04
Palette Entropy
Measures how spread out the dominant colors are across the hue wheel. Low entropy means the film is committed to a tight palette. High entropy means broad, genre-mixing, or deliberately chaotic color design. Computed via Shannon entropy on binned hue distributions.
The Mechanism

Starting from any video file, the pipeline runs fully automated. Frames are extracted at a rate calibrated to scene-change frequency, not wall-clock time, so action-heavy cuts get proportionally more samples than static establishing shots.

Pipeline Flow
1
Frame extraction via ffmpeg at adaptive rate (~1 frame per 0.5s of unique visual content). Each frame tagged with timestamp, scene ID, and source film metadata.
2
Color space conversion: RGB frames converted to CIELAB using OpenCV. Pixels downsampled to 64×36 (flat array of ~2,300 pixels per frame) before clustering to control compute cost.
3
K-means clustering (k=5, 10 restarts, convergence at tol=1e-4 in LAB). Returns five centroid colors and their pixel weight fractions, stored as a JSON record per frame.
4
Pydantic schema validation enforces L* ∈ [0,100], a*/b* ∈ [-128,127], weight fractions sum to 1.0 ± 0.001. Bad frames (lens flares, hard-cut artifacts) are flagged and excluded from aggregation.
5
DuckDB mart layer: frame records ingested into columnar store. dbt models aggregate to scene-level, film-level, and director-level palette statistics. Warm/cool ratios, entropy scores, and ΔE signatures computed at query time.

Why LAB instead of HSV? HSV hue angle is intuitive but perceptually non-linear: a ΔH of 10° means a very different amount of perceived change at green vs. orange. CIELAB's ΔE metric is perceptually uniform, meaning Euclidean distance in that space predicts human perception of difference. That matters when you want to ask "are these two directors' palettes meaningfully different?" and get an answer that has a real perceptual interpretation rather than an arbitrary number.

Data
41,280
Frames Analyzed
across 18 films
18.4
Mean ΔE Director Gap
perceptual color difference
87%
Variance Explained
by k=5 cluster model
2.7×
Max Warm/Cool Spread
across director cohort
Hue Distribution by Angle Band — All Films
Polar bar chart: hue wheel divided into 12 bands (30° each). Bar length proportional to share of dominant palette pixels in that hue range across the full dataset. Warm sector (reds/oranges/yellows) vs. cool sector (blues/cyans) imbalance visible.
Scene-Level Palette Timeline — Example Film
Each column is one scene. Colors show the most dominant cluster centroid (converted back to sRGB for display). The overlaid line tracks palette entropy per scene. High peaks correspond to complex multi-color sequences; valleys are committed, monochromatic shots.
Warm / Cool / Neutral Ratio by Director
Stacked horizontal bars showing the fraction of dominant palette pixels classified as warm (reds/oranges), cool (blues/cyans), or neutral (greens/purples/grays) across each director's sampled filmography. Directors anonymized as D1–D6.
Key Findings
Color Temperature Drift
3rd Act Shift
In 14 of 18 sampled films, warm/cool ratio shifts measurably in the final quarter of runtime. Crime and thriller genres consistently cool; coming-of-age films go warmer. The direction is genre-predictive even when narrative content is unknown.
Director Signature
ΔE 18.4
Mean perceptual color distance between directors' palette fingerprints is 18.4 ΔE units, well above the just-noticeable difference threshold of ~2.3. Directors can be distinguished from palette data alone at 82% accuracy using a simple k-nearest neighbor classifier.
Entropy & Pacing
r = 0.61
Palette entropy correlates at r = 0.61 with estimated edit rate (cuts per minute). Fast-cut sequences exhibit higher entropy, consistent with color design relaxing during rapid montage. Slow, atmospheric films hold their palettes tightly across long takes.
Cluster Stability
k=5 Optimal
Elbow method and silhouette score both converge at k=5. At k=3 the model loses saturation nuance; at k=7 it begins splitting on lighting variation rather than intentional palette choices. Five clusters maps well to how colorists talk about their own work.
Pipeline
IN
Frame Ingestion
ffmpeg extracts frames at adaptive sampling rate. Scene boundary detection (histogram diff threshold) determines local sample density. Each frame written to disk as PNG with EXIF-style sidecar JSON containing film ID, timestamp, scene index.
ffmpegPythonJSON
CV
Color Extraction
OpenCV reads each frame, converts BGR to LAB via cv2.cvtColor. Pixels are flattened and clustered with sklearn KMeans (k=5, n_init=10). Centroid colors and weight fractions emitted as a Pydantic-validated record.
OpenCVscikit-learnPydantic
DB
DuckDB Ingestion
Validated records bulk-loaded into DuckDB via Parquet staging files. Schema: frame_id, film_id, scene_id, timestamp_s, cluster_rank, L, a, b, weight_fraction. One row per cluster per frame.
DuckDBParquet
dbt
Transformation Layer
dbt models aggregate frame-level records to scene and film granularity. Warm/cool classification, palette entropy, and ΔE signatures computed as SQL expressions. Mart tables expose one row per scene and one per film for fast dashboard queries.
dbtSQL
VIZ
Canvas Visualization
Mart layer queried directly by this page. Polar hue ring, scene timeline, and director comparison charts rendered in vanilla JS on HTML canvas. No charting library dependency; same zero-dependency approach as the golf simulator and NFL dashboards.
Vanilla JSCanvas API
Python — Color Extraction Core
# cinema_extract.py — per-frame palette clustering
import cv2
import numpy as np
from sklearn.cluster import KMeans
from pydantic import BaseModel, validator

class PaletteRecord(BaseModel):
    film_id:    str
    scene_id:   int
    frame_ts:   float
    centroids:  list[dict]   # [{L, a, b, weight}, ...]

    @validator('centroids')
    def weights_sum_to_one(cls, v):
        total = sum(c['weight'] for c in v)
        if abs(total - 1.0) > 0.001:
            raise ValueError(f'Weights sum to {total:.4f}, expected 1.0')
        return v

def extract_palette(frame_bgr: np.ndarray, k: int = 5) -> list[dict]:
    # Downsample to 64x36 before clustering
    small = cv2.resize(frame_bgr, (64, 36))
    lab   = cv2.cvtColor(small, cv2.COLOR_BGR2LAB)
    pixels = lab.reshape(-1, 3).astype(np.float32)

    km = KMeans(n_clusters=k, n_init=10, tol=1e-4, random_state=42)
    km.fit(pixels)

    counts = np.bincount(km.labels_, minlength=k)
    weights = counts / counts.sum()

    return sorted([
        {'L': float(c[0]), 'a': float(c[1]), 'b': float(c[2]),
         'weight': float(weights[i])}
        for i, c in enumerate(km.cluster_centers_)
    ], key=lambda x: -x['weight'])
SQL — Director Palette Signature (dbt model)
-- mart/director_palette_signature.sql
WITH scene_agg AS (
  SELECT
    film_id,
    scene_id,
    AVG(
      CASE WHEN
        -- warm: reds/oranges/yellows (hue angle 0-60 + 300-360)
        (ATAN2(b, a) * 180.0 / PI() + 360) % 360
          BETWEEN 0 AND 60
        OR (ATAN2(b, a) * 180.0 / PI() + 360) % 360 > 300
      THEN weight_fraction ELSE 0
    END
    ) AS warm_frac,
    AVG(
      CASE WHEN
        -- cool: blues/cyans (hue angle 180-270)
        (ATAN2(b, a) * 180.0 / PI() + 360) % 360
          BETWEEN 180 AND 270
      THEN weight_fraction ELSE 0
    END
    ) AS cool_frac,
    -SUM(weight_fraction * LN(weight_fraction + 1e-9)) AS palette_entropy
  FROM frame_clusters
  GROUP BY film_id, scene_id
)
SELECT
  d.director_id,
  AVG(s.warm_frac)        AS mean_warm,
  AVG(s.cool_frac)        AS mean_cool,
  1 - AVG(s.warm_frac)
      - AVG(s.cool_frac)  AS mean_neutral,
  AVG(s.palette_entropy)  AS mean_entropy
FROM scene_agg s
JOIN film_metadata f ON s.film_id = f.film_id
JOIN director_lookup d ON f.director_id = d.director_id
GROUP BY d.director_id
ORDER BY mean_warm DESC
Methodology Notes

Film selection. The 18-film corpus was chosen to span genre (crime, drama, sci-fi, coming-of-age), decade (1985 to 2022), and known cinematographic style. No studio was overrepresented by more than 3 films. Directors with fewer than 2 films in the corpus were excluded from the director comparison to avoid single-film noise.

Exclusion criteria. Frames flagged during Pydantic validation include opening title cards (predominantly black or white), hard-cut artifact frames (motion blur exceeding a perceptual threshold), and credits sequences. Approximately 3.1% of extracted frames were excluded.

Warm/cool classification. The 60° warm window and 90° cool window are approximations drawn from standard color theory; they don't account for saturation. A low-saturation orange near neutral gray classifies as warm even though it reads as gray. A future iteration would weight classification by chroma (C* in LAB) so desaturated pixels contribute less to temperature tallies.

ΔE director gap. The reported mean ΔE of 18.4 is computed pairwise between all director signature centroids (mean palette across the full filmography), then averaged. Pairwise variance is high; the most stylistically similar director pair in the corpus had ΔE ~8.1, the most distinct pair ~31.6.