ACT3 bills generation in credits, and before running a scene, a beat or a whole act, the app shows a quote. The quote and the charge had been written separately, so they drifted: the quote missed cast images that were not generated yet, and in one path the prerequisite steps were counted twice when the job actually ran.
I rebuilt the quote to walk the same flow the pipeline walks: it honours Skip and Force, prices every missing prerequisite, is batch-aware, and itemises the result for the breakdown panel. The prerequisite fold now happens only on the quote path, which removed the double-billing, and there are tests on the aggregation rules.
If users see a number before they commit, that number has to come from the code that will charge them.
I expected the biggest jump in AI-WORKER's merge rate to come from a stronger coding model. It came from two structural changes: a reviewer agent that reads every PR before a human does, and a constraint ledger that scores each explicit requirement pass or fail per task.
Swapping models moved quality a little. Adding gates moved it a lot. A weaker model behind a good gate beats a stronger model that can merge whatever it likes.
Omni-Ledger and Kiln let each tenant use their own domain. The customer points a DNS record at our edge, and the site should just work over HTTPS. Caddy's on-demand TLS does exactly that: it issues a certificate the first time a hostname is requested.
Left alone, that is also an open door. Anyone can point any hostname at your IP and make your server request certificates until you hit the issuer's rate limits. The fix is Caddy's ask hook: before issuing, Caddy calls our API, which answers yes only for domains a tenant has registered and verified. Kiln adds an interval and burst limit on top.
After the certificate, forward_auth asks the same API which tenant owns the host and stamps that onto the request as headers, so the apps behind the edge never parse hostnames themselves.
The first version of the Ottr assistant ran on OpenClaw with a system prompt around twenty thousand characters. On an M1 Mac Mini with 16 GB, gemma3:1b took tens of seconds per reply. Swapping to a bigger model made it slower, not better.
Cutting the prompt to about five hundred characters and injecting skills only when a request needed them brought replies under one second on the same hardware and the same model. Prompt length was the whole problem.
Rule I now use: on local inference, measure prompt tokens before touching quantisation or hardware. The cheap fix is usually the prompt.
Baccpacc's 2021 backend was Express and MongoDB with four years of undocumented behaviour. The target was Go on Postgres. Going straight there meant learning what the code did and how to say it in Go at the same time.
I ported to NestJS first. It looked like wasted work. It pinned down the contract, exposed the odd behaviours in a language I could read fast, and the Go rewrite then took weeks instead of months. The NestJS version was thrown away, and it was still the right step.
Netcup gives far more cores and NVMe per rupee than the big clouds, which matters on client budgets. What it does not give you is security groups. There is no network layer in front of the box, so whatever a container publishes is on the internet.
Docker makes that worse than it looks, because published ports bypass ufw's rules by default. The rule I now apply everywhere: ufw allows only 80 and 443, and every service in every compose file binds to 127.0.0.1. Postgres, Redis and the API are reachable from nginx or Caddy on the same host and from nowhere else.
An online checkout needs to hold stock while a customer pays. The obvious design is a reserved flag or a reserved count on the inventory row. It breaks the moment a counter sale and a web reservation touch the same item, because two writers now own one number.
In Omni-Ledger every change is an append-only row: RECEIVE, SALE, TRANSFER, ADJUSTMENT, WASTAGE, and RESERVE and RELEASE, which move on-hand to reserved with no net change. Levels are derived from the log, never edited.
It costs nothing to write and it answers every 'why is stock wrong' question later, because the answer is a row with a reference and a reason.
Episodes are stitched from scenes, and scenes from clips that came from different models, each with its own sample rate, loudness and idea of where audio starts. Fixing sync at the episode level meant chasing drift that had already accumulated across twenty scenes.
Moving normalisation into scene generation, so every scene leaves with the same loudness and a clean start, fixed both the out-of-sync and the missing-audio reports. The same lesson applied to video in 2026: the pipeline used to assume 16:9, and now each scene and episode detects the majority aspect ratio of its source clips before assembly.
Amadeus web services are a chain of eight or more SOAP calls with server-side session state. When a ticket fails to issue, the useful question is which call in which session returned what, and by then the session is gone.
The single most valuable feature in the Sincere Travels backend is an audit log of every Amadeus request and response, exposed in the dashboard. It closed more support tickets than any booking feature. Build the log before the feature.
When we added Veo 3, we sized concurrency the way we had for our own GPUs: more workers, more throughput. Veo does not care how many workers you have. It has its own quota, and past it requests fail in ways that look like our bugs: wrong durations, missing thumbnails, workers stuck on a path that never appears.
The October fixes all came back to one change: concurrency became a per-provider setting read by the worker, not a property of the worker pool. Later, batch generation for Veo 3, Sora 2, Grok and Wan went in on the same rule, so a batch is split to fit each provider's limit rather than the number of machines we happen to be running.
Assembling an episode is a long chain of ffmpeg calls: normalise, concatenate, mux audio, cut a thumbnail. Any one of them can hang on a malformed frame from a generative model, and a hung ffmpeg holds a worker indefinitely.
Every ffmpeg call now runs with a timeout sized to the input duration, and thumbnail extraction has its own retry with a different seek point, because the frame at second one is often the one the model got wrong. Small changes, but they turned a class of 'stuck episode' tickets into a retry that nobody notices.
The ACT3 job runner puts every video, render and audio job in a Temporal workflow. The first month in production, most of our incidents were workflows that never finished: a GPU pod went away, a provider stopped answering, and the workflow sat in Running forever, holding a slot and hiding the failure.
The fix was to treat timeouts as part of the design. Every activity got a start-to-close timeout sized to the job type, the workflow got an overall limit, and a separate monitor workflow finds anything timed out, failed or terminated, marks the job in the app, and restarts or releases it. A scheduler reconciles job status by run ID every minute, which also caught a bug where two workers could pick the same job.
Temporal gives you durability for free. It does not decide how long is too long. That part is yours, per activity.
Blender renders, ComfyUI outputs, downloaded clips and intermediate videos all land on disk. Nobody thinks of that as state until a machine fills up at 3 AM and every job on it fails with an error that mentions nothing about space.
Across the job runner and the visual-automation pipeline I made cleanup part of the job lifecycle: episode files are removed once the final video is uploaded, a sweep deletes working folders older than four hours, shared storage replaced per-machine copies where several workers read the same assets, and Loki's own storage got a retention limit after it became the thing filling the disk.
If the pipeline creates it, the pipeline deletes it. A cron job someone forgets about is not cleanup.