ACT3 AI — distributed video job runner
Temporal.io system orchestrating 22 job types for AI video generation, 3D rendering, compositing and assembly across cloud and local GPUs.
Background
ACT3 turns screenplays into animated episodes. Each shot may need video generation from one of several models, a Blender render, upscaling, compositing and audio. The job runner is the service that does the heavy work for the main platform.
The problem
Jobs run for minutes to hours, cost real money per attempt, and fail in ways an HTTP request cannot handle: a provider times out at minute eight, a GPU pod disappears, ffmpeg hangs on a corrupt frame. The earlier queue-based approach lost state on worker restarts and retried whole jobs from scratch, which doubled spend.
The provider landscape also changed monthly. Veo, Sora, Grok, Wan and Seedance each had different limits, pricing and failure modes, and the product wanted to route between them without redeploying.
Approach
- Put Temporal at the centre. Every job is a durable workflow. Each step is an activity with its own retry policy and timeout. A worker restart resumes exactly where it stopped.
- Split scheduling from execution. A scheduler decides whether a job goes to a RunPod GPU or a local machine based on job type, queue depth and cost. Workers are managed by PM2 and packaged as Docker images per job family.
- Wrap each video provider behind one interface with provider fallback, so a Seedance failure can retry on another model without human intervention.
- Harden from production incidents: ffmpeg timeouts and thumbnail retry, panel sheet decomposition for storyboard frames, dynamic RunPod scaling that spins pods down when the queue drains.
- Instrument everything. Prometheus, Loki and Promtail behind nginx give per-job-type latency, failure rate and GPU utilisation.
- Run it as separate processes: a master that schedules, workers (slaves) that execute, a logger and a deployer, each with its own PM2 ecosystem file and setup scripts for Linux, macOS and Windows machines.
- Treat timeouts as design: a monitor workflow finds timed-out, failed and terminated runs and releases or restarts them; a scheduler reconciles job status by run ID every minute; a credit-refresh scheduler keeps billing in step.
- Automate the GPU side: RunPod pod setup and health checks, ComfyUI generation alongside the hosted models, and character motion-capture jobs.
Architecture
Outcome
- In production since 2025 as the rendering backbone of the ACT3 platform. Seedance became the most-used provider once fallback made it safe to lead with the cheapest model.
- Worker restarts and deploys no longer lose in-flight jobs.
- Top contributor on the repository across a team of five.
Lessons
- Durable execution is worth the operational cost of running Temporal. The alternative is rebuilding half of it badly inside a queue consumer.
- Provider abstractions should expose cost and latency, not just success. Routing decisions need both.