Building Stoke: An Open-Source, Self-Hosted AI Dev Platform

Project Overview

Stoke (github.com/souravdutt/stoke-ai, formerly named Forge until a naming collision forced a rename) is a self-hosted AI platform I built and open-sourced under TechGeeta: it provisions GPU compute on Vast.ai, deploys open-weight coding models, and wires the resulting endpoint into an editor as a bring-your-own-key backend for GitHub Copilot, Continue.dev, and Cline. The goal was concrete, not aspirational: replace my own daily use of hosted coding assistants with a self-hosted model I control, at a fraction of the cost, provisioned and torn down on demand.

Stack: Python, Typer CLI, FastAPI (local dashboard), React/TypeScript (dashboard frontend), llama.cpp, Vast.ai API, Cloudflare Tunnel Status: Daily-driver mode — the maintainer's own stoke.techgeeta.com deployment is in regular use; open-sourced under MIT

Architecture, Decided Before It Was Needed

The platform is organized around three layers — compute (where), models (what), capabilities (what I need) — and every non-trivial decision along the way is written down as an ADR before implementation, not after. Eleven of them exist so far, covering things like why a database was deliberately ruled out until Phase 4 (state lives in a plain StateStore abstraction instead), why inference engines sit behind a formal EngineBase interface rather than being hardcoded to llama.cpp, and why the CLI resolves a request like ai coding down to a specific model/engine/quantization instead of making the caller specify all of it upfront. Architecture is explicitly frozen once a phase ships — new abstractions need a demonstrated need, not a hunch, to get added.

Two Capabilities, One Optional Shared-Instance Optimization

ai coding runs a persistent llama.cpp server for editor integration; ai video runs a separate torch-based batch workflow. Independently, each provisions and bills its own Vast.ai instance — correct, but wasteful for a single person who mostly uses one and occasionally the other, since the idle capability still bills. ADR-007 adds an opt-in shared-instance mode: one instance can host both, with only one model loaded into GPU memory at a time, switched by config, under a VRAM budget — without touching how either capability behaves when run alone. That's the kind of constraint (billing waste on idle GPU-rental infrastructure) that only shows up once you're actually paying for the thing you built, not something you'd invent designing on a whiteboard.

Cutting Cold-Start Time With a Public Binary Cache

The default path for a new GPU architecture is compiling llama.cpp from CUDA source — a 10-15 minute wait before a single token comes back. Stoke ships a public, read-only HuggingFace dataset cache of precompiled llama-server bundles keyed by GPU architecture (cuda86 = RTX 3090/3080 Ti/A5000/A6000, etc.), so a first deploy on a common consumer GPU installs in about 60 seconds instead. No token or auth is needed to read it; a contributor with an uncached architecture just falls back to the slower source build automatically, no error, no manual step.

A Local Dashboard That Wraps the CLI, Not a Second Implementation

The platform was CLI-only for a while — search Vast.ai offers, deploy, edit capabilities/*.yaml by hand, tail logs across terminal tabs. ai dashboard is a local-only FastAPI app (binds 127.0.0.1, not part of the public tunnel) that wraps the existing orchestrator/provider/registry functions directly instead of reimplementing any of that logic against a second surface — so the dashboard and the CLI can never drift into disagreeing about deployment state, because they're calling the same functions underneath a thin API layer and a React frontend.

The Eval Framework — What It Actually Is

This is the part worth being precise about, because it's easy to overstate. eval/ runs benchmark tasks (eval/tasks/phase3/, ~11 realistic coding tasks — Laravel N+1 fixes, PHP type bugs, Vue race conditions, PR review, and similar) against a live deployment over an SSH tunnel, capturing time-to-first-token, tokens/sec, and GPU utilization alongside the raw model response. eval/scorer.py then walks a human through each task's criteria interactively in the terminal — pass/fail per criterion, with notes — and eval/compare.py diffs the resulting scores across model runs.

That's a real, working model-benchmarking and comparison tool. It is not an automated or CI-gated eval suite — there's no LLM-judge scorer, and the only GitHub Actions workflow in the repo lints/builds/tests the dashboard's React frontend, not the eval pipeline itself. If I were pitching this as "agent eval harnesses that gate a build," that would overstate what's actually wired up. What it demonstrates honestly is: benchmark task design, a structured scoring rubric, cross-run comparison tooling, and the judgment to know exactly where automation stops and a human is still in the loop — which is a meaningfully different (and smaller) claim than "CI-gated agent evals," and I'd rather state the smaller true thing than the bigger one that doesn't hold up to a clone-and-read.

What's Genuinely Missing

No agent observability stack (no Langfuse/LangSmith/OTel-style tracing — there's no multi-step agent here to trace, just single-shot completions), no data lineage/provenance tooling, and no CI wired to the Python test suite (one integration test file covers the video workflow runtime with stubbed nodes; the orchestrator, capability loader, and cost ledger have no tests yet). None of that is dressed up as done.

What I'd Highlight in a Technical Interview

The shared-instance mode is the one I'd go deepest on — it's a real answer to "how do you keep a self-hosted GPU platform from bleeding money on idle capacity," discovered by actually running the thing and watching a bill, not designed in the abstract. The second is the ADR discipline itself: eleven decisions, each written down with the context and the alternative considered, before the code that implements them. For a codebase built with heavy AI-assistant involvement, that written trail is the actual mechanism that keeps a fast-moving, AI-assisted codebase reviewable months later — not a process box being checked.

Tech Stack: Python, Typer, FastAPI, React/TypeScript, llama.cpp, Vast.ai, Cloudflare Tunnel Role: Founder & Sole Engineer, TechGeeta (Open Source) Status: Daily-driver mode, MIT licensed Repo: github.com/souravdutt/stoke-ai