Building an AI Video Clipping Pipeline: Architecture, Tradeoffs, and What We Learned

Building an AI Video Clipping Pipeline: Architecture, Tradeoffs, and What We Learned Processing video at scale is one of those problems that looks straightforward until you're in production at 3 AM watching a queue pile up. This is the actual architecture behind ClipSpeedAI — what we built, why we made the tradeoffs we did, and what we'd do differently. The Core Challenge The product is simple to describe: take a YouTube URL, return 8-12 vertically reformatted short-form clips with captions and virality scores, in under 15 minutes. The technical reality is a multi-stage async pipeline with three fundamentally different processing environments — JavaScript/Node.js for orchestration and API serving, Python for machine learning inference, and FFmpeg for video encoding — all needing to coordinate without stepping on each other. Stage 1: Job Ingestion and Queue Management Every video job enters through a REST API endpoint that validates the URL, creates a job record in Supabase, and pushes