Stopping one site's database queries from slowing down every other site

Simple Host · resource isolation prototype · for SIM-335 and the Thin Server paper

Short answer. One site sending heavy queries makes today's Simple Host much worse for everyone else. In the test, the other sites' slowest-1% latency went from 8 ms to 290 ms, and three quarters of their requests were refused. A cap of one running query per site fixes most of this: p99 drops to 35 ms and nothing is refused. The change is about 20 lines. A full process-per-site sandbox, started on demand and evicted LRU, did best (p99 16 ms) and costs about 3 ms to cold-start a site. That is worth building later, not first.

1. The problem, in plain words

Simple Host is a single program that serves every site. Each site has its own small SQLite database, and visitors' pages can send it read-only SQL. Every query runs inside that one program, on the same CPUs, memory and disk.

A single SQL statement can be enormously expensive and still be legal. Examples: a table joined to itself and filtered with LIKE, a recursive query that counts to 200 million, or a DISTINCT over a self-join that spills hundreds of megabytes to temp files. Rate limits count requests, not how much each one costs. Five requests a second within the limit can still take all the CPU, so a restaurant menu on another site loads slowly or fails.

2. What Simple Host does today

From main at 9283ccf0 (22 Oct 2026). The running binary carries the same knobs (SITE_STORAGE_ACQUIRE_WAIT_MS is present in /usr/local/bin/simple-host).

ControlWhereValueWhat it bounds
Driverinternal/handler/site_storage_sql.go:23ncruces/go-sqlite3, in-processNothing on its own. Every site shares one process.
Global SQL gatesite_storage_sql.go:515-536runtime.NumCPU() slots (4 on this box), wait 250 ms (config/limits.go:421-422)Total queries running at once. Not per site, so one site can hold every slot.
Query timeout + interruptsite_storage_sql.go:185-200Visitor query 1 s, visitor write 2 s, owner 5 s (limits.go:424-426). Interrupt goes through conn.SetInterrupt(ctx), which the driver polls from a progress handler.Wall-clock time per query. That is not CPU, and it gets slower under contention.
Memory ceilingsite_storage_sql.go:26-28, :194256 MiB per connection (4096 * 65536)Memory per query, but too high to matter. Four slots × 256 MiB = 1 GiB.
Temp storagenot set (no temp_store anywhere in main)SQLite default: temp files on diskNothing. Big sorts spill to disk without limit (448 MB in one 10 s query).
Length limitssite_storage_sql.go:209-210SQL 32 KB, value 1 MBStatement size, not cost.
Authorizersite_storage_sql.go:270-303No ATTACH, PRAGMA, temp objects or transactions; visitor writes only per policyWhat a query may touch, not what it costs.
Rate limitssite.go:325-326, limits.go:415Per IP and per visitor; visitor storage burst 60, then 5/sRequest count only.
Instruction budget, per-site cap, CPU/IO quotasnone——

3. Options

MechanismCPU?Memory?Disk IO?Per site?Overhead
Shorter wall-clock timeout (1 s → 100 ms)Partly. Caps each query, but four at once still fill the box.NoPartly, through shorter runsNoNone
Instruction budget (progress handler counts VM ops)Partly. Deterministic per query, independent of load.NoPartlyNo, unless the budget is per siteAbout 0. The driver already runs a progress handler.
Per-site concurrency cap (1 query in flight per site)Yes, in practice. One site gets at most one slot.Indirectly (one × the per-query cap)IndirectlyYesOne channel per active site
Lower memory cap plus temp_store=MEMORY (32 MB)NoYes, per queryYes: sorts can no longer spill to diskPer connectionNone. Big sorts fail fast with "out of memory".
Process per site in its own cgroup (on demand, LRU)Yes, enforced by the kernel (cpu.max)Yes (memory.max)Yes (io.max available; not tested)YesCold start 3 ms p50 / 17 ms p99, about 1.4 MB per warm site, one pipe hop per query

4. Test setup

The four heavy queries, run alone (10 s timeout)

Query256 MB cap (today)8 MB cap1 MB cap32 MB, temp in memory
Recursive CTE10 s CPU, timed outsamesamesame
Cross join LIKE5.4 s, completessamesamesame
DISTINCT sort spill10.9 s, timed out, 448 MB written to disktimed out, 511 MB writtenout of memory at onceout of memory in 0.4 s, 0 MB written
Full-scan sort5.2 s, 45 MB written5.2 s, 14 MB writtenout of memory at once4.9 s, 0 MB written

A lower memory cap on its own does not stop disk spill, because SQLite sends temp data to files outside the cap. Only temp_store=MEMORY brings spill under the cap. Also, the driver panics on out-of-memory rather than returning an error (go-sqlite3 conn.go:493-494). Simple Host would rely on net/http's per-request recover, so that needs an explicit recover before the cap is lowered.

5. Results: the other 199 sites while one site attacks

Scenariop50 msp99 msOK req/s (of 300)RefusedPeak memDisk written
Quiet: no attacker, no limits (baseline)1.57.62980%15 MB55 MB
Attacker, no limits at all15.5832970%525 MB561 MB
Attacker, today's Simple Host (4 global slots, 1 s timeout)1682907874%57 MB688 MB
Today's settings on a 1-CPU box (1 slot)2132531595%70 MB484 MB
Today + timeout 100 ms711622970%25 MB198 MB
Today + instruction budget (500k VM ops)1582692758%48 MB607 MB
Today + 1 query in flight per site1.7352990%63 MB558 MB
Per-site cap + 100 ms + budget (combined)1.8352990%21 MB386 MB
Combined + 32 MB cap, temp in memory2.0492990%21 MB54 MB
Process per site, cgroup 25% CPU / 48 MB each, 32 warm, LRU1.2162990%83 MB137 MB

"Refused" means the well-behaved request got a "storage is busy" response from the global gate. "Disk written" is the whole process, mostly the attacker's temp-file spill. Repeats agree within about 10%, except the no-limit run (p99 74 vs 92 ms).

What the numbers say

6. The on-demand + LRU sandbox

Each site that receives a query gets a small worker process. The worker runs in its own cgroup (cpu.max 25000 100000, meaning a quarter of a CPU; memory.max 48M; no swap) with GOMAXPROCS 1, GOMEMLIMIT=24MiB and one warm connection to its database. The main process sends it queries over a pipe. At most 32 workers stay warm, and when a new site needs one, the least-recently-used idle worker is killed.

MeasuredValue
Cold start (fork + cgroup + open DB + ready)3.1 ms p50 · 17 ms p99
Memory per warm workerabout 1.4 MB (cgroup memory.current)
Starts / evictions in 20 sabout 2,120 / 2,090
Warm hit rate with 32 warm, 200 sites, Zipf 1.165%

This is a deliberately bad case: 200 active sites competing for 32 warm workers, so a third of requests paid a cold start. Even so, p99 was the best of any scenario. At about 1.4 MB each, 500 warm workers would fit in about 700 MB, which pushes the hit rate toward 100% for a real long-tail workload. The worker re-uses the same binary in -mode worker, so there is nothing new to ship. The test did not use io.max or seccomp.

7. Recommendation and build order

  1. Per-site in-flight cap in acquireStorageSQLite (site_storage_sql.go:520). Take a per-site token before the global slot: 1 for visitors, 2 for owners, and wait 250 ms. About 20 lines plus a test. This gives most of the win. Under attack, p99 goes from 290 to 35 ms and refusals go from 74% to 0.
  2. Cut the visitor query timeout to 100–250 ms (SITE_STORAGE_VISITOR_QUERY_TIMEOUT_MS, a config change) and add an instruction budget by counting progress-handler calls in the interrupt context. A budget is deterministic, so a query that passes when the box is idle also passes when it is busy.
  3. Bound memory and disk: set PRAGMA temp_store=MEMORY server-side before installing the authorizer, and lower storageSQLiteMaxMemory from 256 MiB to 32 MiB. First, add an explicit recover around statement execution, because the driver panics on out-of-memory. Map it to a clean 413/422 "query too expensive" response. Check that no existing site needs more than 32 MB per query.
  4. Later: the process-per-site sandbox with on-demand start and LRU eviction, behind a flag. Build it when there are untrusted or paying tenants at scale, or when steps 1–3 still show interference in production. It adds kernel-enforced CPU and memory limits, contains crashes (an out-of-memory panic kills one worker, not the server), and gives the best tail. The cost is process management, a pipe hop, and a few milliseconds of cold start.

Caveat on step 1: one owner with many sites could still use several slots at once. A follow-up can cap per account as well as per site. A legitimately busy site is also serialised to one query at a time. Its cheap queries take about 1 ms, so it still has about 1,000 queries/s of headroom.

8. What this changes in the Thin Server paper

Files

Source and raw results: workspace-2/eb1/paper/sandbox-test/. That folder holds main.go, run.sh, results/*.json (two repeats per scenario, plus pilot/) and results/attacks*.jsonl. Linear: SIM-335 (paper related-work and measurements). Prototype results, 5 Nov 2026; not production measurements.