SPIELBERG: how I produce video with code using Remotion, an agent and human approvals

Why video as code

A video made with code is closer to a web page than to an editing session. Every frame is a function of time: same inputs, same output. That brings three things a traditional editor doesn't:

1. Repeatability. Changing one line of the script and re-rendering doesn't break anything that was already right.
2. Versioning. The video lives in Git: it can be diffed, reviewed and reverted like any other code.
3. Automation. An agent can prepare scenes, captures, voice and subtitles, because everything is files and commands.

SPIELBERG started out making reels for a client with a simple chain (Remotion, Whisper and a config file) and grew into a monorepo with three parts: the video engine, a CLI of about 580 lines and a set of Claude Code skills that coordinate the work.

Before producing: four questions

The first lesson came early. I asked for a horizontal piece and an external skill delivered a vertical one, outside the project and outside Remotion. Since then, nothing is produced without answering four questions:

• Orientation and resolution: vertical, horizontal or square, and at what size.
• Frames per second.
• Where it is delivered: a social network, a website, a presentation.
• Whether it is a standalone piece or a layer inside another edit.

It looks like bureaucracy; in practice it is what prevents redoing an entire render because someone pictured a different format.

The pipeline: from brief to MP4

Each video lives in its own folder, with the script, a manifest and the Remotion composition. The route is always the same, even when techniques change:

1. Script by scenes. A custom format with video and scene blocks that both the agent and I can read.
2. Scene decisions. A TypeSafe model proposes, for each scene, which global layers it carries (narrator, music, subtitles) and what goes in the foreground and what in support. In the first test it got 8 of 9 scenes right. Nothing is applied without my OK.
3. Animatic with audio. A white sketch of each scene, with the voice already playing, to judge pacing before investing in finish.
4. Hashed approvals. Contract, script, storyboard and animatic are approved with my literal reply, tied to a hash. If the script changes afterwards, the approval expires on its own.
5. Layered production. Remotion is always the master timeline. Every other source comes in as a layer: procedural film, software captures with a virtual cursor, stock with its license manifest, 3D, maps, charts, particles or background removal with Apple Vision.
6. Audio and subtitles. Voice, local transcription and a mix with deterministic ducking.
7. Render and verification. Pre-flight check, render and audio quality control on the encoded MP4, before copying it to the delivery folder.

One gallop, eight languages

The best demonstration of why Remotion is in charge is a 32-second piece: a horse gallops without stopping while the world that draws it changes style eight times (silhouette, flat vector, cartoon, pixel art, pencil, articulated cut-outs, stylised 3D and realistic). Each change arrives with a curtain the horse crosses nose first, at 120 beats per minute and one stride per beat. The gallop follows Muybridge's 1878 photographs, with a single skeleton for all eight styles.

The first seven styles are drawn with code; the realistic closing scene is the only AI-generated shot (Kling, through Higgsfield), and I asked for it explicitly.

Mixing 2D illustration, 3D geometry and generated video in one sequence, with transitions that match to the frame, is exactly the kind of work that gets fragile in a traditional editor and here is just another composition.

More pieces already coming out of the engine

Brand closers. Two vertical pieces for social media: the LeonelKrea outro, a golden chess knight that turns into the logo, and a seed that grows into a sunflower and ends on the brand's cat.

LeonelKrea outro (9 s) and blooming seed (5 s), at 1080×1920.

Product video. To pitch a management software for an oil camp, the video tells the problem (spreadsheets that duplicate and arrive late), the proposal (a management dashboard plus tools for each team), the camp's model as the interface and a phased plan. It has voice-over and subtitles, and came out in Spanish, English and Italian from the same project.

The proposal video, almost two minutes long. Interfaces are a concept and the data is sample data.

Real 3D when it's needed. Not everything is drawn with code: this robot is modelled, lit and animated in Blender, and goes into a piece like any other clip.

Determinism: the rule in charge

The engine has seven master rules, and the first is absolute: nothing depends on the clock. No Math.random, no Date.now and no CSS animations; everything is computed from the current frame. Render a frame twice and it comes out identical.

That discipline has performance consequences I learned by measuring:

• Don't clone canvases. A trail effect over a blurred panel created twelve canvases at 1080×1920 and took between 125 and 835 ms per frame, when the budget at 24 fps is 41.7 ms. Cloning canvas or WebGL is now prohibited.
• Real shaders. Of 204 React Bits components audited, 91 are WebGL shaders. They need ANGLE in headless Chromium and time tied to the frame; faking them with SVG is banned. With that, still frames with shaders render in under 400 ms.
• Measurable precision. In a mixed test piece, five out of five audio cues land within 6.7 ms of the picture.

Audio gets verified too

The voice comes from ElevenLabs and transcription runs locally with whisper.cpp, accelerated on the Mac's chip. But an automatic transcription is never the final subtitle: it is checked against the script, because the script is the source of truth.

Subtitles were the best example of measuring before having opinions. The old rule (four words or 28 characters per block) produced 59 blocks in one piece: 12 single words and 19 under one second. I rewrote it following professional subtitling standards (two lines of up to 42 characters, between 1 and 7 seconds, up to 17 characters per second) with a dynamic-programming cut. Result: 20 readable blocks.

Quality control measures loudness, true peak, silences and spectrum on the encoded file, not on the mix.

That check exists because the encoder fooled me once: 128 kbps AAC overshot up to 4.8 dB on transients, not the 0.2 dB the documentation considered fine, and the MP4 clipped. Since then, verification runs on the file that gets delivered.

Local review: the human tweaks without overwriting the agent

The last stretch isn't code. Double-clicking the video document opens a small macOS app with the Remotion player: there I fix a subtitle, raise or lower voice and music in decibels and export. My tweaks are saved in a controls file with revisions, separate from what the agent writes, so neither overwrites the other.

Local review: mix in dB, subtitle size and editable text that keeps timings and voice.

The character in that capture comes from another piece of the system: a procedural 2D character creator (three hairstyles, three faces and three outfits, 27 combinations) with lip sync so it speaks with my voice.

What costs money, and what doesn't

Generative AI is part of SPIELBERG, but as an add-on, not as the engine. Video and image models and paid voice or music are only used when I ask, and the CLI requires explicit confirmation before any billable call.

The rule has a story. A video model returned a 502 error, the agent retried and both requests were charged. It wasn't much money (real spend on those tests was 1.35 dollars), but the pattern mattered: after a 5xx error from a paid service, never retry automatically.

In the same spirit, licenses are treated like code: 29 music tracks arrived with an MIT-licensed repository, and a code license doesn't prove music rights, so they were quarantined.

What is still missing

SPIELBERG is in daily use, but it isn't finished, and I'd rather say so with the same precision as the rest:

• Portability. The full system still has to be tested on another computer with a complete regression run.
• Calibration. The fast-edit features are technically complete, but still need listening tests, calibration on three real productions and performance checks.
• The review app depends on this repository and on Node; it isn't distributed or notarized.
• A general-purpose editor is still a parked idea: today the chosen route is Remotion controls plus the small app.

If you want to see how SPIELBERG fits with the rest of the system, in AURORA I explain how I keep an eye on the agents that run it, and in Dexter how I specify before letting an agent touch the code.

Frequently asked questions