How We Built an AI Video-Clipping Engine with Whisper and LLMs

Every content team we talk to has the same asset rotting in a folder: hours of long-form video — webinars, podcasts, livestreams, course recordings — that everyone agrees "should be cut into shorts" and nobody has time to cut. Done by hand, one hour of footage takes an editor several hours: scrub the whole thing, find the moments that stand alone, cut them, subtitle them word by word. At that cost, the clips simply don't get made, and the footage stays a sunk cost instead of a distribution engine.

We built an automated clipping engine to solve this for our own products — it runs today as part of our subscription AI tools platform, turning long recordings into subtitled, social-ready short clips with no human in the loop. This post is the honest architecture writeup: the three-stage pipeline, the infrastructure decisions that surprised us, and where automation still loses to a human editor. Not a tutorial, but enough detail that a technical reader could sanity-check their own build.

The pipeline: transcribe, select, render

Stage 1 — Transcription with Whisper

Everything downstream depends on knowing what was said and exactly when. We run faster-whisper — an optimized implementation of OpenAI's open-source Whisper model — on our own compute, which gives us word-level timestamps at a fraction of the cost of per-minute transcription APIs. That cost structure matters when users upload hour-long files: transcription priced per minute of audio eats subscription margin alarmingly fast, while self-hosted Whisper turns it into a fixed hardware cost you amortize.

Two hard-won details. First, timestamps are the product here, not just the text: subtitle timing and clip boundaries are only as good as the alignment, so we validate word timing rather than trusting it blindly — long silences and music segments can make timestamps drift. Second, our audience speaks Mandarin as well as English, and Whisper's multilingual accuracy is genuinely good but not uniform; mixed-language speech (very common in Taiwan) is the stress case we test hardest.

Stage 2 — Highlight selection with an LLM

This is the stage that was impossible five years ago. The timestamped transcript goes to an LLM — we use Claude — with a job that is more editorial than technical: read the whole talk, find the segments that stand alone. A good clip is not just an interesting sentence; it needs a self-contained arc — a hook, a payoff, and no dangling reference to "what I said earlier" — and it must open strong, because short-form viewers grant you about two seconds.

What we learned making this reliable:

• Structured output or nothing. The model returns candidate segments as strict JSON — start time, end time, a hook line, a confidence score — validated against a schema. Freeform "here are some good moments!" output is unusable in a pipeline.
• Boundaries need snapping. LLM-proposed cut points are approximate; we snap them to sentence boundaries and natural pauses from the transcript timing, because a clip that opens mid-word reads as broken no matter how good the content is.
• Selection criteria are a product decision. "Find highlights" is underspecified. Contrarian claims? Emotional peaks? Concrete numbers? We encode an explicit editorial rubric in the prompt, and tuning that rubric changed output quality more than any model upgrade did.
• Cost is controllable. Transcripts are long, so this stage is token-heavy — exactly the kind of workload where model tiering and prompt structure pay off. We covered those tactics in our LLM cost-control writeup; this pipeline is where several of them were learned.

Whisper gives you what was said, the LLM decides what was worth saying, and ffmpeg does the honest labor of making it watchable.

Stage 3 — Rendering with ffmpeg

The least glamorous stage and the largest share of engineering time. For each selected segment we cut the source video, reframe for vertical where needed, and burn in word-timed subtitles styled for sound-off viewing — because most short-form video is watched muted, subtitles are not an accessibility extra, they are the primary interface. All of it is ffmpeg, orchestrated by a Python/FastAPI service. Anyone who has styled burned-in CJK subtitles — font fallback, line-breaking rules that differ from English, safe margins across aspect ratios — knows where the weeks went. It works; it was never elegant.

The architecture decision that mattered most: separate the heavy compute

Video is a different species of workload from a web app. Transcription and encoding will happily pin every CPU core for minutes at a time; put that on the same box as your website and every customer feels it. So the engine runs as an isolated service on dedicated compute, and the customer-facing platform never touches it directly: the user-facing app owns accounts, billing, quotas, and uploads, then proxies jobs to the engine over an internal API. The browser never sees the engine at all.

This split bought us three things. Operationally, a runaway encode cannot degrade the storefront. Commercially, the engine stays a pure computation service — metering and paywalls live in one place, in the product layer. And architecturally, everything became a queue: users expect "we'll notify you when your clips are ready," not a progress spinner, which means the whole pipeline optimizes for throughput and graceful retry instead of latency. Async-by-default is, frankly, the correct shape for almost every heavy AI workload we have built since.

Where automation still loses

Anti-hype section, from operating this in production:

• The model can't see. Selection runs on the transcript, so a purely visual moment — a demo, a whiteboard, a facial reaction — is invisible to it. Talk-driven content (interviews, lectures, podcasts) automates beautifully; visually-driven content does not.
• Taste has a ceiling. The engine reliably produces good clips from good source material. The single transcendent clip a great human editor finds — the pause, the look, the cut on exactly the right beat — is not something we would claim to match. The honest pitch is volume and consistency at near-zero marginal cost, not artistry.
• Source quality is a hard floor. Crosstalk, bad mics, and heavy background music degrade transcription, and every downstream stage inherits the damage. No prompt fixes bad audio.

What this generalizes to

Strip away the video specifics and the shape is reusable: perception model → LLM judgment with structured output → deterministic rendering, run async on isolated compute, metered at the product layer. We have seen the same skeleton fit document processing, audio content, and image workflows. If you are sitting on a content backlog — or a repetitive judgment-plus-processing workflow that looks like this — it is exactly the kind of thing worth pressure-testing as a scoped build; that conversation starts at our contact page.

The takeaway: an automated clipping pipeline is no longer research — Whisper, a well-instructed LLM, and ffmpeg get you to production, and the real difficulty lives in boundary-snapping, subtitle rendering, and infrastructure isolation rather than in "the AI part." Know what it can't do, architect the heavy path async, and the footage folder stops being a graveyard.