Building Demoick: A Deterministic Auto-Zoom Engine and Headless GPU Compositor

Project Overview

Demoick is a macOS desktop app, built as founder and sole engineer at TechGeeta, that turns a raw screen recording into a cinematic product demo automatically — auto-zoom on clicks and typing, auto-speed on idle stretches, local transcription and voice-over, no manual editing required. It's explicitly modeled on Screen Studio's behavior, and the product spec is blunt about what actually matters: "the auto-zoom engine is the heart of the product — without it the product has no value." Everything else in the app exists to support that one decision loop.

Stack: Tauri 2, Rust, React, TypeScript, Zustand, Tailwind, wgpu (Metal), ffmpeg, whisper.cpp, ONNX Runtime Status: In active development, ad-hoc signed (no notarized release yet)

The Pipeline

A recording becomes an export through four distinct stages, and keeping them separate is deliberate — when the output looks wrong, the first question is which stage owns the bug:

recording (.mp4 + cursor/click sidecar)
  → analysis:   cursor path, click events, typing episodes, scene cuts
  → editing:    speed ramps, idle removal, target duration      → EditDecision
  → camera:     click/typing episodes → per-frame camera pose   → RenderManifest
  → render:     GPU compositor draws each frame                 → output .mp4

The Camera Engine Is Deterministic, Not Spring Physics — On Purpose

Early in the project there was a spring-simulated "director" engine with tuning knobs for camera feel. It's gone. Not deprecated-but-present — actually deleted, because it was "fully superseded and never actually reachable from any live code path." What ships instead samples a fixed ease-in/hold/ease-out pose from a plan built off the interaction stream:

// Each click, typing run, or still span becomes one ZoomElement.
// Adjacent elements close enough in time glide directly between
// origins instead of zooming out and back in — that reads as a stutter.
struct ZoomElement { start_ms: u64, end_ms: u64, cx: f32, cy: f32, zoom: f32, origin: Origin }

fn sample_at(elements: &[ZoomElement], source_ms: u64, cfg: &ZoomConfig) -> CameraPose {
    // fixed ease-in/hold/ease-out — no spring simulation, no per-frame physics
}

Typing gets a top-left-anchored origin that eases toward the caret as it nears the frame edge, so the camera stays legible without a jarring pan. Clicks get a centered origin. Sustained idle relaxes back to 1x. It's a smaller, more predictable design than a spring simulator, and it's the one that's actually reachable from every code path — export, live preview, and the Timeline's manual overrides all sample the same function.

A Hand-Rolled GPU Compositor, Not a Video Editing Library

Rendering is a headless wgpu (Metal) compositor — one fullscreen fragment shader does the entire look: background gradient, drop shadow, the rounded-rect inset recording (sub-pixel camera sampling — Catmull-Rom bicubic when zoomed in, bilinear otherwise, motion blur integrated across camera motion), click ripple, then cursor glow/ring/spotlight. The uniform struct on the Rust side has to stay byte-for-byte in sync with the WGSL shader's own struct — std140 layout rules mean every vec2 has to sit on an 8-byte boundary, which is why the struct carries explicit scalar padding instead of a convenient vec3. Get that wrong and the shader silently reads garbage into the wrong field; there's no compiler that catches a layout mismatch across a Rust/WGSL boundary.

Compose and encode overlap through double-buffered async readback — the GPU composites the next frame while the previous one is still being written out — with a CPU crop-and-scale fallback behind a RENDER_CPU=1 flag for machines or bugs where the GPU path doesn't apply.

Webcam picture-in-picture is a second, fully isolated GPU pass rather than folded into the main shader — a deliberate boundary so a webcam rendering bug can't corrupt the primary recording's pixels. It composites into the same output with alpha blending after the main pass has already written its pixels, and a missing or broken webcam file is treated as advisory (logged, skipped) rather than failing the whole render — the same posture the pipeline takes toward missing mic or system audio.

Fully Local AI: Transcription and Voice-Over

Subtitles come from on-device Whisper transcription (whisper.cpp) with voice-activity detection ahead of it and a styleable animation/attach layer after — no audio ever leaves the machine. Voice-over is a two-tier local TTS picker: a Standard tier (Kokoro-82M, ONNX, runs in-process) that's always available, and an opt-in Expressive tier (Chatterbox, adjustable emotion, 23 languages) gated behind a RAM check for machines that can afford it. The Standard tier's 54-voice roster isn't guessed — it's a compile-time constant cross-checked directly against the actual model repo's file listing, specifically so the install flow only has to fetch audio weights, never a voice manifest that could drift out of sync with what the model actually ships.

The Analysis Cache Has a Documented Bug History

Every analysis run is cached by content hash (sha256 of the source file) under an explicit pipeline version number, and that version's changelog is kept inline in the cache module itself — not a separate changelog file that goes stale. It reads like a small bug ledger: v3 fixed a seek bug that returned the same keyframe for every sample; v5 added hard-cut scene detection for window switches; v7 added the keystroke-cadence and accessibility-focus event stream the camera engine now depends on. Bumping the version number is how a fix gets force-invalidated across every previously-cached recording, instead of quietly leaving stale results in place.

What's Deliberately Not Finished

The project's own architecture doc is specific about what's incomplete, and I'd rather represent that accurately than round it up:

• Typing detection is weak — real typing often gets misclassified as idle, which is the single biggest lever on the zoom engine actually firing when it should.
• Decode, not rendering, is the throughput bottleneck (~9–15 fps) — the GPU compositor isn't what's slow.
• Webcam background blur and AI background removal are wired end-to-end (persisted, exposed in the UI, threaded to the shader) but inert — both need a person-segmentation model the app doesn't ship yet.
• No CI — Rust and TypeScript tests exist (unit tests on the Rust side, Vitest on the frontend) but nothing runs them on push yet.
• No notarized release — without a paid Apple Developer account, the built app is ad-hoc signed and Gatekeeper quarantines it on download; the README documents the manual quarantine-clear step.

What I'd Highlight in a Technical Interview

The webcam-as-isolated-second-pass decision is the one I'd go deepest on — it's a concrete example of choosing a slightly more expensive architecture (two render passes instead of one shader that does everything) specifically to bound the blast radius of a bug. The second would be deleting the spring-physics camera engine rather than leaving it behind a flag "just in case" — dead code that's merely unreachable is still a maintenance liability, and removing it was the correct call once the simpler deterministic sampler covered every real call site.

Tech Stack: Tauri 2, Rust, React, TypeScript, Zustand, Tailwind, wgpu (Metal), ffmpeg, whisper.cpp, ONNX Runtime Role: Founder & Sole Engineer, TechGeeta Duration: 2025 - Present Status: In active development