When the Cursor team published their engineering deep-dive on **Continuity** and **Origin**, their custom Git hosting engine, it sparked widespread discussion across systems engineering circles. Instead of managing fleets of stateful Linux virtual machines attached to persistent NVMe drives, Cursor turned AWS S3 into a high-throughput, distributed write-ahead log (WAL) using atomic compare-and-swap (CAS) commits. The architectural premise was compelling: treat Git repositories as immutable packfile streams in cloud object storage, use S3 `If-Match` ETags for linearizable consensus, and scale reads horizontally through sub-10ms conditional HTTP 304 cache validations. Most people read the post as an interesting peek into how an AI code editor handles monorepos at scale. I saw an opportunity to challenge traditional infrastructure conventions. I decided to replicate Cursor's architecture from the ground up. I started by validating the core distributed storage primitives, benchmarked atomic CAS under concurrent race conditions, and then pushed the boundary: I built **Strata Git**, an internet-facing developer platform featuring native Git CLI Smart HTTP support, an in-browser repository explorer, stateless session security, and deployed the entire platform serverless on AWS Lambda with genuine **$0.00/month idle costs**. Here is the complete engineering journey, the distributed algorithms, and the lessons learned building it. --- ## 1. The Spark: Cursor's Continuity and the Flaws of Stateful Git Git is fundamentally an append-only directed acyclic graph (DAG) of cryptographically hashed objects: blobs, trees, commits, and tags. Yet, virtually every modern Git hosting platform (GitHub Enterprise, GitLab Gitaly, self-hosted Gitea, or Bitbucket) is architected around stateful Linux machines running on persistent block storage (EBS or local SSDs). This legacy model introduces three systemic engineering bottlenecks: 1. **Storage Lock-In and IOPS Contention:** Every repository is pinned to a specific filesystem volume. Scaling storage requires resizing block devices, running background disk-rebalancing daemons, and constantly over-provisioning IOPS to survive concurrent CI/CD clones. 2. **Brutal Idle Economics:** Stateful instances must run 24 hours a day, 7 days a week, consuming CPU and memory while waiting for occasional developer pushes. For internal developer platforms or side projects, running reliable high-availability Git clusters costs hundreds of dollars each month while sitting 95% idle. 3. **Fragile High-Availability (HA) Topology:** Replicating Git repositories across availability zones traditionally requires distributed volume mirroring or complex Raft-based state machines (like GitLab Gitaly Cluster). When a primary node dies during a push, reconciling partially written refs and packfile state is notoriously error-prone. Cloud object storage (AWS S3 or Cloudflare R2) fundamentally solves durability: it delivers 99.999999999% (11 9s) durability, automatic multi-AZ redundancy, virtually infinite scale, and sub-$0.02/GB monthly costs. The missing piece has always been **linearizable concurrency control**. Git requires strict atomic updates: branch pointers (`refs/heads/main`) must only advance if the parent commit matches what the client saw before pushing. Without a database or a POSIX file lock, how do you prevent split-brain pushes on S3? --- ## 2. The Core Foundation: S3 as a Write-Ahead Log Strata stores all repository data in AWS S3 across two primary object classes: ``` s3://git-at-any-scale/ ├── repos/ │ └── milangohel/vaani-ledger/ │ ├── wal_index.json <-- Linearizable Manifest (Version & Branch Tips) │ └── packs/ │ ├── 3b1a9f02.pack <-- Immutable Git Packfile (Push 1) │ ├── 3b1a9f02.idx <-- Git Pack Index │ ├── 7c8e2d14.pack <-- Immutable Git Packfile (Push 2) │ └── 7c8e2d14.idx └── _auth/ └── auth_manifest.json <-- Users, Scoped PATs, ACL Policies ``` ### The WAL Index (`wal_index.json`) The single authoritative source of truth for a repository is its manifest object. It records: - An incrementing integer version number (`version: 42`). - The canonical commit SHA for each reference (`refs/heads/main`, `refs/tags/v1.0`). - An ordered chronological log of active packfile objects that comprise the repository state. ```json { "repoId": "milangohel/vaani-ledger", "version": 3, "defaultBranch": "main", "refs": { "refs/heads/main": "7a941bf28059b58793b868bb2b89d424b9176311", "refs/heads/feature": "9c12e5a410b0d39e87518428394e1d67812903ab" }, "packfiles": [ "packs/base_mega_pack.pack", "packs/delta_push_2.pack", "packs/delta_push_3.pack" ], "updatedAt": "2026-10-04T12:00:00.000Z" } ``` ### The `unpackLimit = 1` Packfile Trick Under standard behavior, when you push a small commit to a Git server, Git explodes the push payload into individual loose object files (`.git/objects/4b/825dc...`). On object storage, writing 50 loose files would require 50 distinct S3 PUT operations, destroying write performance and inflating costs. Strata forces Git core into immutable batch mode: ```bash git config receive.unpackLimit 1 git config transfer.unpackLimit 1 ``` With `unpackLimit = 1`, Git preserves the incoming push stream as a single, self-contained, compressed `.pack` file and an accompanying `.idx` file. The server uploads this single binary blob directly to `packs/.pack` in S3 with a single streaming PUT request. --- ## 3. Distributed Consensus via S3 Atomic CAS How do we commit the new packfile and update branch references without Postgres, Redis, or distributed locks? We use **Atomic Compare-And-Swap (CAS)** implemented through S3 conditional writes. S3 natively supports the HTTP `If-Match` header. When you pass `If-Match: ""` on a `PutObject` call, S3 guarantees that the write will succeed **only** if the object in S3 currently matches that exact ETag. Here is the exact TypeScript implementation from our storage engine: ```typescript import { S3Client, PutObjectCommand, GetObjectCommand } from "@aws-sdk/client-s3"; export interface WalIndex { repoId: string; version: number; defaultBranch: string; refs: Record; packfiles: string[]; updatedAt: string; } export class S3WalEngine { constructor( private s3: S3Client, private bucket: string ) {} /** * Fetches the latest WAL manifest along with its current S3 ETag version. */ async getWalIndex(repoId: string): Promise<{ index: WalIndex; etag: string }> { const key = `${repoId}/wal_index.json`; const res = await this.s3.send( new GetObjectCommand({ Bucket: this.bucket, Key: key, }) ); const bodyString = await res.Body!.transformToString("utf-8"); const index: WalIndex = JSON.parse(bodyString); const etag = res.ETag!.replace(/"/g, ""); // Normalize ETag return { index, etag }; } /** * Commits an updated WAL manifest using S3 Atomic CAS. * Throws an error if another node committed changes concurrently. */ async commitWalIndex( repoId: string, updatedIndex: WalIndex, expectedEtag: string ): Promise { const key = `${repoId}/wal_index.json`; updatedIndex.updatedAt = new Date().toISOString(); try { const res = await this.s3.send( new PutObjectCommand({ Bucket: this.bucket, Key: key, Body: JSON.stringify(updatedIndex, null, 2), ContentType: "application/json", IfMatch: `"${expectedEtag}"`, // Atomic precondition enforced by S3 }) ); return res.ETag!.replace(/"/g, ""); } catch (err: any) { // S3 returns HTTP 412 Precondition Failed if the ETag changed if (err.name === "PreconditionFailed" || err.$metadata?.httpStatusCode === 412) { throw new Error(`CONCURRENCY_RACE: Repository ${repoId} was updated concurrently.`); } throw err; } } } ``` ### What Happens During Concurrent Pushes? Imagine Developer A and Developer B push to `main` at the exact same millisecond: ```mermaid sequenceDiagram autonumber actor A as Dev A actor B as Dev B participant S3 as AWS S3 Storage Note over A,B: Concurrent Git push to main A->>S3: Read wal_index.json (ETag: "ABC") B->>S3: Read wal_index.json (ETag: "ABC") A->>S3: Upload Packfile A B->>S3: Upload Packfile B A->>S3: PUT wal_index.json (If-Match: "ABC") B->>S3: PUT wal_index.json (If-Match: "ABC") S3-->>A: HTTP 200 OK (ETag becomes "DEF") Note over A: Accepted (Fast-Forward) S3-->>B: HTTP 412 Precondition Failed Note over B: Rejected (Fetch First) ``` 1. Both developers read `wal_index.json` with `ETag: "ABC"`. 2. Developer A's commit reaches S3 first. S3 matches `If-Match: "ABC"`, writes the new file, and rotates the ETag to `"DEF"`. 3. Developer B's commit arrives milliseconds later with `If-Match: "ABC"`. S3 checks the current state, sees `"DEF" != "ABC"`, and rejects the write with **HTTP 412 Precondition Failed**. 4. The server downloads the fresh index `"DEF"`. If Developer B's commit is not a clean fast-forward of A's new tip, Git CLI immediately informs Developer B: `[rejected] - fetch first`. No distributed lock manager, no Postgres row-level locks, no Redis key expirations. Consensus is achieved purely via HTTP conditional semantics on object storage. --- ## 4. Scaling Reads: Sub-10ms Cache Hits & Cold Materialization In Git workloads, read operations (`git clone`, `git fetch`, CI builds, file browsing) outnumber write operations by at least 50 to 1. If every read operation had to parse S3 indexes from scratch, latency would cripple the developer experience. Strata implements horizontal replica caching using standard HTTP cache validation. When a replica node or serverless instance handles a `git fetch`, it holds a locally cached copy of `wal_index.json` and makes a conditional S3 request: ```typescript export async function checkReplicaFreshness( s3: S3Client, bucket: string, repoId: string, cachedETag: string ): Promise<{ modified: boolean; freshIndex?: WalIndex; newETag?: string }> { try { const res = await s3.send( new GetObjectCommand({ Bucket: bucket, Key: `${repoId}/wal_index.json`, IfNoneMatch: `"${cachedETag}"`, // Sub-10ms hot-path check }) ); // If we reach here, the repository has new commits (HTTP 200) const body = await res.Body!.transformToString("utf-8"); return { modified: true, freshIndex: JSON.parse(body), newETag: res.ETag!.replace(/"/g, ""), }; } catch (err: any) { // S3 returns 304 Not Modified if the ETag matches if (err.$metadata?.httpStatusCode === 304) { return { modified: false }; } throw err; } } ``` ### The Performance Outcome: - **Bandwidth Consumed:** Exactly 0 bytes. - **S3 Egress Cost:** $0.00. - **Roundtrip Latency:** ~8ms to 12ms. When changes are detected, the node inspects the new `packfiles` array, identifies the single newly added delta packfile, and downloads only those missing bytes. ### Ephemeral Cold Materialization ("Cattle, Not Pets") We also tested the ultimate resilience property: booting up an empty container with zero persistent disk, running `materialize()`, and reconstructing a complete, operational Git repository from S3 in ~300ms. Because Git repositories are self-assembling collections of packfiles, a node simply downloads the active packs, runs `git init --bare`, and sets the reference tips from `wal_index.json`. --- ## 5. Amortized Compaction: Solving Packfile Sprawl Write-ahead logs have an inherent operational cost: append-only growth causes fragment sprawl. If developers push 50 small commits, the repository is fragmented across 50 `.pack` files. Running `git fetch` across 50 indexes increases index lookups and degrades CPU throughput. Cursor addressed this with amortized compaction. I implemented this as a dedicated compaction worker: ```mermaid flowchart LR A["10+ Fragmented
Packfiles in S3"] --> B["Consolidate & Merge
git repack -ad"] B --> C["Single Optimized
Mega-Packfile"] C --> D["Atomic CAS Swap
wal_index.json"] D --> E["Garbage Collect
Old Deltas"] ``` ```typescript export async function runAmortizedCompaction( repoDir: string, repoId: string, walEngine: S3WalEngine, s3: S3Client, bucket: string ): Promise { const { index, etag } = await walEngine.getWalIndex(repoId); // Trigger compaction only if fragment count exceeds threshold if (index.packfiles.length < 10) { return; } // 1. Run native Git delta-repack on the bare repository scratch space await Bun.spawn(["git", "-C", repoDir, "repack", "-a", "-d", "-f", "--depth=50", "--window=250"]).exited; // 2. Locate the freshly generated unified packfile const objectsDir = `${repoDir}/objects/pack`; const files = await readdir(objectsDir); const newPackName = files.find((f) => f.endsWith(".pack"))!; const newIdxName = files.find((f) => f.endsWith(".idx"))!; // 3. Stream the unified packfile to S3 under an immutable hash key const megaPackKey = `packs/compacted_${Date.now()}_${newPackName}`; const megaIdxKey = megaPackKey.replace(".pack", ".idx"); await s3.send(new PutObjectCommand({ Bucket: bucket, Key: `${repoId}/${megaPackKey}`, Body: Bun.file(`${objectsDir}/${newPackName}`), })); await s3.send(new PutObjectCommand({ Bucket: bucket, Key: `${repoId}/${megaIdxKey}`, Body: Bun.file(`${objectsDir}/${newIdxName}`), })); // 4. Atomically point the WAL manifest to the single mega-packfile via CAS const compactedIndex: WalIndex = { ...index, version: index.version + 1, packfiles: [megaPackKey], updatedAt: new Date().toISOString(), }; await walEngine.commitWalIndex(repoId, compactedIndex, etag); } ``` The beauty of this design: **CPU compaction work is done once on the server**. Every replica node and developer machine simply downloads the pre-computed mega-packfile on their next fetch, keeping client performance consistently fast. --- ## 6. Stateless Routing via Highest Random Weight Hashing In a multi-node cluster, how do you route developer requests to the right node without maintaining a centralized routing database or running ZooKeeper? To achieve this, we implemented **Highest Random Weight (HRW) Rendezvous Hashing**: ```typescript import { createHash } from "node:crypto"; export function rankNodesForRepo(repoId: string, activeNodes: string[]): string[] { return [...activeNodes].sort((nodeA, nodeB) => { const scoreA = computeScore(repoId, nodeA); const scoreB = computeScore(repoId, nodeB); return scoreB - scoreA; // Descending rank }); } function computeScore(repoId: string, nodeId: string): bigint { const hash = createHash("sha256") .update(`${repoId}:${nodeId}`) .digest("hex") .slice(0, 16); return BigInt(`0x${hash}`); } ``` Given any repository name (e.g. `milangohel/vaani-ledger`), every node independently and deterministically calculates the exact same ranking of servers. - If Node 1 is highest-ranked, it handles the push. - If Node 1 crashes, every other node instantly agrees that Node 2 is the new primary with **zero leader elections, zero split-brain risk, and zero consensus latency**. --- ## 7. Going Serverless: Pure Git Smart HTTP on AWS Lambda Cursor's post focused on monorepos hosted on internal Kubernetes clusters. I wanted to see if this architecture could power an open, internet-facing developer platform with **zero servers running at idle**. I packaged the Bun runtime, the native Git core engine, and the **AWS Lambda Web Adapter (LWA)** into an optimized, multi-stage Docker container deployed to AWS Lambda: ```mermaid flowchart LR Dev["Developer Terminal
git clone / push"] -->|"Stream HTTPS"| LURL["Lambda
Function URL"] LURL -->|"Port 3000"| LWA["Lambda Web
Adapter"] LWA --> Bun["Bun CGI
Smart HTTP"] Bun --> Tmp["/tmp Bare
Scratch"] Bun --> S3["AWS S3 Bucket
(WAL + Packs)"] ``` ### Native Git Smart HTTP over Serverless When Git communicates over HTTP, it uses two CGI endpoints: 1. `GET /:repo/info/refs?service=git-upload-pack` (Reference advertisement) 2. `POST /:repo/git-upload-pack` or `POST /:repo/git-receive-pack` (Object exchange) AWS Lambda traditionally buffers entire request payloads in memory before passing them to handlers. Git packfiles can easily be hundreds of megabytes. Using the **AWS Lambda Web Adapter** and enabling `RESPONSE_STREAM` on the Lambda Function URL, Strata streams the incoming binary packfile directly into an ephemeral bare repository mounted on `/tmp/repos`, captures the generated packfile, streams it immediately to S3, and commits the CAS manifest. When the push completes, the local `/tmp` directory is cleanly recycled. If no developer pushes code for the next three days, **zero CPU cycles run and zero dollars are billed**. --- ## 8. Eliminating the Database: Stateless Auth and PATs Most web architectures rely on PostgreSQL for user tables and Redis for session caches. Adding a database cluster would instantly defeat the purpose of a zero-idle-cost serverless platform. Strata eliminated databases entirely using three architectural primitives: | Layer | Zero-Database Primitive | Implementation | | :--- | :--- | :--- | | **User & Token Store** | S3 Atomic CAS Manifest | Encrypted user records & scoped PATs | | **Web Session State** | Stateless HMAC-SHA256 Cookie | Sub-millisecond CPU signature verification | | **CLI Authentication** | Argon2id Scoped PATs | Fine-grained read/write Git token validation | ### 1. The S3 Auth Manifest (`_auth/auth_manifest.json`) All registered users, argon2id password hashes, scoped Personal Access Tokens (`pat_...`), and repository visibility rules live inside an encrypted S3 manifest. Updates to users or token revocation happen via the exact same Atomic CAS engine. ### 2. Stateless HMAC-SHA256 Session Cookies When users sign in via the web explorer, the server issues a cryptographically signed cookie: ```typescript export class SessionManager { private static SECRET = process.env.SESSION_SECRET || "fallback-secret-key"; static sign(payload: SessionPayload, ttlSeconds: number = 604800): string { const header = Buffer.from(JSON.stringify({ alg: "HS256", typ: "JWT" })).toString("base64url"); const exp = Math.floor(Date.now() / 1000) + ttlSeconds; const data = Buffer.from(JSON.stringify({ ...payload, exp })).toString("base64url"); const signature = createHmac("sha256", this.SECRET) .update(`${header}.${data}`) .digest("base64url"); return `${header}.${data}.${signature}`; } static verify(token: string): SessionPayload | null { const parts = token.split("."); if (parts.length !== 3) return null; const [header, data, signature] = parts; const expected = createHmac("sha256", this.SECRET) .update(`${header}.${data}`) .digest("base64url"); if (signature !== expected) return null; // Tamper detection const payload = JSON.parse(Buffer.from(data, "base64url").toString("utf-8")); if (payload.exp && payload.exp < Math.floor(Date.now() / 1000)) return null; // Expiry return payload; } } ``` Every serverless invocation verifies the cookie signature in microseconds using CPU math: zero network requests, zero database connections, and zero cold-start connection pool exhaustion. ### 3. Identity Verification in GitHub OAuth We also addressed a critical account association challenge: if account linking matches solely on username handles (`login === "torvalds"`), someone could register `torvalds` with an unverified password, and subsequent GitHub logins would inherit that pre-registered account. We resolved this by enforcing strict primary verified email matching: ```typescript // Auto-link accounts ONLY if the GitHub email is verified AND matches the existing account const verifiedEmail = ghUser.emailVerified && ghUser.email ? ghUser.email.toLowerCase().trim() : undefined; if (verifiedEmail) { const existingByEmail = Object.values(manifest.users).find( (u) => u.email && u.email.toLowerCase().trim() === verifiedEmail && !u.githubId ); if (existingByEmail) { existingByEmail.githubId = String(ghUser.id); existingByEmail.githubUsername = ghUser.login; await this.saveManifest(manifest); return existingByEmail; } } // If username is already claimed by someone with a different email, assign a clean, collision-free handle let targetUsername = ghUser.login.toLowerCase().trim(); if (manifest.users[targetUsername]) { targetUsername = `${targetUsername}-gh`; } ``` --- ## 9. Building the Web Explorer: "Mini-GitHub" Developers need more than command-line endpoints: they need to browse code, view commit logs, inspect diffs, and generate access keys. I implemented a server-side rendered (SSR) web interface directly within the Bun engine, following high-density Linear-style visual standards: - **Zero Client Bundles:** Pure server-rendered HTML and CSS: loads in single-digit milliseconds with zero client-side JavaScript hydration lag. - **Repository Tree & Blob Viewer:** Explores directories, renders files with syntax formatting, and serves raw files directly. - **Commit History Timeline:** Visualizes commit chains with relative timestamps, author avatars, and SHA badges. - **In-Browser PAT Management:** Authenticated users generate `pat_...` keys directly in the browser at `/settings/tokens`, choose granular scopes (`read`, `write`, `admin`), and revoke keys with a single click. --- ## 10. Architecture Comparison | Dimension | Traditional Git (GitLab/Gitaly) | Cursor Continuity (Internal) | Strata Git (Our Implementation) | | :--- | :--- | :--- | :--- | | **Primary Storage** | Stateful NVMe / EBS Block Volumes | AWS S3 Object Storage | AWS S3 (Standard + Express) | | **Concurrency Model** | PostgreSQL row locks or Redis | S3 `If-Match` Atomic CAS | S3 `If-Match` ETag Atomic CAS | | **Compute Topology** | 24/7 Dedicated Virtual Machines | Long-running Kubernetes pods | Serverless AWS Lambda Containers | | **Idle Infrastructure Cost** | $60 to $300+ / month | Significant cluster baseline | **$0.00 / month (Scale to Zero)** | | **Database Dependency** | PostgreSQL + Redis required | Distributed KV Store | **Zero external databases** | | **Git Protocol Support** | Native Smart HTTP & SSH | Internal gRPC / RPC | Standard Git CLI Smart HTTP | | **Hot Read Cache Latency** | Local filesystem I/O | S3 304 validation (~10ms) | S3 304 validation (<10ms) | | **Developer Web Portal** | Monolithic Ruby/Go web UI | Internal proprietary editor UI | Built-in SSR Minimalist Web UI | --- ## 11. Production Performance & Concrete Metrics We benchmarked Strata Git deployed live on AWS Lambda (`eu-north-1`) backed by an AWS S3 bucket: - **Idle Infrastructure Cost:** Exactly **$0.00 / month**. If no pushes occur, zero compute charges accrue. - **Cold Start Latency:** **~240ms** (Alpine Linux container + Bun runtime + AWS Lambda Web Adapter). - **Warm Git Push Latency:** **~180ms total roundtrip** for a standard multi-file commit (includes packfile streaming to S3 and Atomic CAS manifest commit). - **Hot Read Check (304 Not Modified):** **~9ms** response time over public HTTPS. - **Automated Test Suite:** **68/68 unit and integration tests** passing in **4.55 seconds** executing against real S3 primitives. > Note: Because Git objects are cryptographically hashed and immutable, packfiles cached on S3 edge storage can be served with perpetual `Cache-Control: public, max-age=31536000, immutable` headers, offloading read egress entirely to CDN edge locations. --- ## 12. 3 Key Architectural Lessons 1. **Object Storage With Conditional Writes Is a Distributed Database:** The industry often treats S3 as a dumb bucket for backups and static assets. With strong read-after-write consistency and atomic `If-Match` preconditions, S3 can serve as an ACID-compliant, highly available write-ahead log for mission-critical distributed systems. 2. **Stateful Protocols Can Run Serverless:** Git was designed in 2005 for long-lived local filesystems. By strictly isolating disk operations to ephemeral scratch memory (`/tmp`) and treating object storage as the authoritative source of truth, even stateful protocols like Git Smart HTTP can run inside stateless, scale-to-zero serverless environments. 3. **Radical Simplicity Destroys Operational Overhead:** By eliminating PostgreSQL, Redis, RabbitMQ, and persistent EBS volumes, you eliminate 90% of the operational surface area. No database connection pool limits to tune, no disk volume resizing alerts at 2 AM, and no cross-region database replication lag. Simplicity is the ultimate scalability feature.

I Built My Own GitHub