Stopping one site's database queries from slowing down every other site
Simple Host · resource isolation prototype · for SIM-335 and the Thin Server paper
Short answer. One site sending heavy queries makes today's Simple Host much worse for everyone else. In the test, the other sites' slowest-1% latency went from 8 ms to 290 ms, and three quarters of their requests were refused. A cap of one running query per site fixes most of this: p99 drops to 35 ms and nothing is refused. The change is about 20 lines. A full process-per-site sandbox, started on demand and evicted LRU, did best (p99 16 ms) and costs about 3 ms to cold-start a site. That is worth building later, not first.
1. The problem, in plain words
Simple Host is a single program that serves every site. Each site has its own small SQLite database, and visitors' pages can send it read-only SQL. Every query runs inside that one program, on the same CPUs, memory and disk.
A single SQL statement can be enormously expensive and still be legal. Examples: a table joined to itself and filtered with LIKE, a recursive query that counts to 200 million, or a DISTINCT over a self-join that spills hundreds of megabytes to temp files. Rate limits count requests, not how much each one costs. Five requests a second within the limit can still take all the CPU, so a restaurant menu on another site loads slowly or fails.
2. What Simple Host does today
From main at 9283ccf0 (22 Oct 2026). The running binary carries the same knobs (SITE_STORAGE_ACQUIRE_WAIT_MS is present in /usr/local/bin/simple-host).
| Control | Where | Value | What it bounds |
|---|---|---|---|
| Driver | internal/handler/site_storage_sql.go:23 | ncruces/go-sqlite3, in-process | Nothing on its own. Every site shares one process. |
| Global SQL gate | site_storage_sql.go:515-536 | runtime.NumCPU() slots (4 on this box), wait 250 ms (config/limits.go:421-422) | Total queries running at once. Not per site, so one site can hold every slot. |
| Query timeout + interrupt | site_storage_sql.go:185-200 | Visitor query 1 s, visitor write 2 s, owner 5 s (limits.go:424-426). Interrupt goes through conn.SetInterrupt(ctx), which the driver polls from a progress handler. | Wall-clock time per query. That is not CPU, and it gets slower under contention. |
| Memory ceiling | site_storage_sql.go:26-28, :194 | 256 MiB per connection (4096 * 65536) | Memory per query, but too high to matter. Four slots × 256 MiB = 1 GiB. |
| Temp storage | not set (no temp_store anywhere in main) | SQLite default: temp files on disk | Nothing. Big sorts spill to disk without limit (448 MB in one 10 s query). |
| Length limits | site_storage_sql.go:209-210 | SQL 32 KB, value 1 MB | Statement size, not cost. |
| Authorizer | site_storage_sql.go:270-303 | No ATTACH, PRAGMA, temp objects or transactions; visitor writes only per policy | What a query may touch, not what it costs. |
| Rate limits | site.go:325-326, limits.go:415 | Per IP and per visitor; visitor storage burst 60, then 5/s | Request count only. |
| Instruction budget, per-site cap, CPU/IO quotas | none | — | — |
3. Options
| Mechanism | CPU? | Memory? | Disk IO? | Per site? | Overhead |
|---|---|---|---|---|---|
| Shorter wall-clock timeout (1 s → 100 ms) | Partly. Caps each query, but four at once still fill the box. | No | Partly, through shorter runs | No | None |
| Instruction budget (progress handler counts VM ops) | Partly. Deterministic per query, independent of load. | No | Partly | No, unless the budget is per site | About 0. The driver already runs a progress handler. |
| Per-site concurrency cap (1 query in flight per site) | Yes, in practice. One site gets at most one slot. | Indirectly (one × the per-query cap) | Indirectly | Yes | One channel per active site |
Lower memory cap plus temp_store=MEMORY (32 MB) | No | Yes, per query | Yes: sorts can no longer spill to disk | Per connection | None. Big sorts fail fast with "out of memory". |
| Process per site in its own cgroup (on demand, LRU) | Yes, enforced by the kernel (cpu.max) | Yes (memory.max) | Yes (io.max available; not tested) | Yes | Cold start 3 ms p50 / 17 ms p99, about 1.4 MB per warm site, one pipe hop per query |
4. Test setup
- Program:
paper/sandbox-test/main.go(903 lines). It mirrorssite_storage_sql.go: a fresh connection per request, read-only URI for reads,DBCONFIG_DEFENSIVE, the same length limits, a write lock per site, and the global gate with a 250 ms wait. - Machine: Oracle ARM VM, 4 vCPU (CPU part
0xd0c, Neoverse N1), 19 GB RAM, Linux 6.17. Go 1.27.1,ncruces/go-sqlite3 v0.35.6(wasm2go). - Confinement: every run is
sudo systemd-run --scope -p CPUQuota=100% -p MemoryMax=1G -p MemorySwapMax=0, then the privileges drop toubuntu(run.sh). That is one CPU's worth of time (cpu.max 100000 100000), with GOMAXPROCS 2. - Data: 200 site databases (WAL), 2,000 rows each (3,000 on the noisy site), 100-byte text bodies, and an index on
(cat, score). - Well-behaved load: 300 requests/s, open-loop Poisson, sites chosen by Zipf (s=1.1), 97% indexed top-10 reads and 3% inserts. Latency is timed from the scheduled arrival, so queueing delay is counted.
- Noisy site: 4 clients send four legal queries back to back: recursive CTE, cross join with
LIKE,DISTINCTsort spill, and full-scan sort. Each one alone costs 5–10+ CPU-seconds. - Runs: 20 s each. Two repeats for every scenario except
today_1cpuandcombined + 32 MB temp-in-memory(one each). The numbers below are means of the repeats.
The four heavy queries, run alone (10 s timeout)
| Query | 256 MB cap (today) | 8 MB cap | 1 MB cap | 32 MB, temp in memory |
|---|---|---|---|---|
| Recursive CTE | 10 s CPU, timed out | same | same | same |
Cross join LIKE | 5.4 s, completes | same | same | same |
DISTINCT sort spill | 10.9 s, timed out, 448 MB written to disk | timed out, 511 MB written | out of memory at once | out of memory in 0.4 s, 0 MB written |
| Full-scan sort | 5.2 s, 45 MB written | 5.2 s, 14 MB written | out of memory at once | 4.9 s, 0 MB written |
A lower memory cap on its own does not stop disk spill, because SQLite sends temp data to files outside the cap. Only temp_store=MEMORY brings spill under the cap. Also, the driver panics on out-of-memory rather than returning an error (go-sqlite3 conn.go:493-494). Simple Host would rely on net/http's per-request recover, so that needs an explicit recover before the cap is lowered.
5. Results: the other 199 sites while one site attacks
| Scenario | p50 ms | p99 ms | OK req/s (of 300) | Refused | Peak mem | Disk written |
|---|---|---|---|---|---|---|
| Quiet: no attacker, no limits (baseline) | 1.5 | 7.6 | 298 | 0% | 15 MB | 55 MB |
| Attacker, no limits at all | 15.5 | 83 | 297 | 0% | 525 MB | 561 MB |
| Attacker, today's Simple Host (4 global slots, 1 s timeout) | 168 | 290 | 78 | 74% | 57 MB | 688 MB |
| Today's settings on a 1-CPU box (1 slot) | 213 | 253 | 15 | 95% | 70 MB | 484 MB |
| Today + timeout 100 ms | 71 | 162 | 297 | 0% | 25 MB | 198 MB |
| Today + instruction budget (500k VM ops) | 158 | 269 | 275 | 8% | 48 MB | 607 MB |
| Today + 1 query in flight per site | 1.7 | 35 | 299 | 0% | 63 MB | 558 MB |
| Per-site cap + 100 ms + budget (combined) | 1.8 | 35 | 299 | 0% | 21 MB | 386 MB |
| Combined + 32 MB cap, temp in memory | 2.0 | 49 | 299 | 0% | 21 MB | 54 MB |
| Process per site, cgroup 25% CPU / 48 MB each, 32 warm, LRU | 1.2 | 16 | 299 | 0% | 83 MB | 137 MB |
"Refused" means the well-behaved request got a "storage is busy" response from the global gate. "Disk written" is the whole process, mostly the attacker's temp-file spill. Repeats agree within about 10%, except the no-limit run (p99 74 vs 92 ms).
What the numbers say
- Today's gate makes an attack worse than having no gate. The attacker keeps all four slots busy for up to 1 s each, and everyone else waits 250 ms and is turned away. With no gate the other sites are slower but served. On a 1-CPU box, 95% are refused.
- The per-site cap does most of the work. Once the attacker can hold only one slot, the other three keep serving. p99 falls from 290 ms to 35 ms and nothing is refused.
- Timeouts and budgets alone do not isolate. They shorten each heavy query, but the attacker sends the next one immediately. An instruction budget counts work, not time, so it is predictable under load. It is cheap insurance, not a fix.
- Only temp-in-memory bounds disk IO. Disk writes fell from 386 MB to 54 MB, about the same as the quiet baseline.
- A process sandbox per site gives the best tail (p99 16 ms) and real kernel limits. It also used less CPU (0.76 cores), because each worker keeps one warm connection instead of opening a fresh one per request.
6. The on-demand + LRU sandbox
Each site that receives a query gets a small worker process. The worker runs in its own cgroup (cpu.max 25000 100000, meaning a quarter of a CPU; memory.max 48M; no swap) with GOMAXPROCS 1, GOMEMLIMIT=24MiB and one warm connection to its database. The main process sends it queries over a pipe. At most 32 workers stay warm, and when a new site needs one, the least-recently-used idle worker is killed.
| Measured | Value |
|---|---|
| Cold start (fork + cgroup + open DB + ready) | 3.1 ms p50 · 17 ms p99 |
| Memory per warm worker | about 1.4 MB (cgroup memory.current) |
| Starts / evictions in 20 s | about 2,120 / 2,090 |
| Warm hit rate with 32 warm, 200 sites, Zipf 1.1 | 65% |
This is a deliberately bad case: 200 active sites competing for 32 warm workers, so a third of requests paid a cold start. Even so, p99 was the best of any scenario. At about 1.4 MB each, 500 warm workers would fit in about 700 MB, which pushes the hit rate toward 100% for a real long-tail workload. The worker re-uses the same binary in -mode worker, so there is nothing new to ship. The test did not use io.max or seccomp.
7. Recommendation and build order
- Per-site in-flight cap in
acquireStorageSQLite(site_storage_sql.go:520). Take a per-site token before the global slot: 1 for visitors, 2 for owners, and wait 250 ms. About 20 lines plus a test. This gives most of the win. Under attack, p99 goes from 290 to 35 ms and refusals go from 74% to 0. - Cut the visitor query timeout to 100–250 ms (
SITE_STORAGE_VISITOR_QUERY_TIMEOUT_MS, a config change) and add an instruction budget by counting progress-handler calls in the interrupt context. A budget is deterministic, so a query that passes when the box is idle also passes when it is busy. - Bound memory and disk: set
PRAGMA temp_store=MEMORYserver-side before installing the authorizer, and lowerstorageSQLiteMaxMemoryfrom 256 MiB to 32 MiB. First, add an explicitrecoveraround statement execution, because the driver panics on out-of-memory. Map it to a clean 413/422 "query too expensive" response. Check that no existing site needs more than 32 MB per query. - Later: the process-per-site sandbox with on-demand start and LRU eviction, behind a flag. Build it when there are untrusted or paying tenants at scale, or when steps 1–3 still show interference in production. It adds kernel-enforced CPU and memory limits, contains crashes (an out-of-memory panic kills one worker, not the server), and gives the best tail. The cost is process management, a pipe hop, and a few milliseconds of cold start.
Caveat on step 1: one owner with many sites could still use several slots at once. A follow-up can cap per account as well as per site. A legitimately busy site is also serialised to one query at a time. Its cheap queries take about 1 ms, so it still has about 1,000 queries/s of headroom.
8. What this changes in the Thin Server paper
- Design. The paper says sites run no server code. Visitors' SQL is a small program that runs on the server, though, and its cost is controlled by whoever writes the query. Design should name a resource-isolation layer next to the access rules. Today's state: a global gate, a wall-clock timeout and a 256 MB memory cap. The target: a per-site in-flight cap, an instruction budget, and temp storage held in a bounded memory cap. The optional process sandbox goes in as future work.
- Related work. This sentence in Section 7 is too strong: "we avoid the need for per-site isolation on the server by having no per-site code to isolate". Replace it with: no per-site code needs isolating for safety, but per-site query cost still needs isolating for fairness, and that is done with cheap in-process limits rather than per-site micro-VMs.
- Discussion / Limitations. Add noisy neighbours as a limitation, with these numbers. Without a per-site cap, one site's legal queries can refuse 74% of other sites' requests on a 4-CPU box (95% on 1 CPU). With the cap, p99 stays within about 5× of quiet (35 ms vs 7.6 ms).
- Security claim (Section 5.4, SIM-367). The "fixed attack surface" argument holds for code execution and secrets. It does not cover availability. Add resource exhaustion through legal SQL as an attack class, marked "mitigated by per-site limits", not "removed by design". Report the result as measured on a prototype, not production.
Files
Source and raw results: workspace-2/eb1/paper/sandbox-test/. That folder holds main.go, run.sh, results/*.json (two repeats per scenario, plus pilot/) and results/attacks*.jsonl. Linear: SIM-335 (paper related-work and measurements). Prototype results, 5 Nov 2026; not production measurements.