Benchmarks

A benchmark is a small program that does one job many times and writes down how long it took. Ferry has a set of them in the repository, so that "is it fast enough?" and "did this change make it slower?" have answers made of numbers instead of feelings. Anyone can run them with one command.

This page says what they measure, how to run them, what to expect, and how the repository uses them on every release.

What they measureCopy Anchor Link

Ferry has three places where time goes: the device, the server, and the work between them. There is a benchmark for each, and each one runs the real code with real stores. Nothing is mocked: the only thing missing is the network, which is replaced by a function call, so the numbers are the cost of Ferry itself and not of your connection.

Words used below:

• Row: one record, a task in the examples.
• Storage: where the device keeps its copy (memory, SQLite, IndexedDB, a key-value store). See Storages.
• Store: where the server keeps the log (SQLite, Postgres). See Stores.
• Bootstrap: a new device downloads everything it may see, once. See Overview.
• Statement: one SQL command the server sends to its database, such as an INSERT. A write that changes several things sends several.
• Outbox: the writes a device made and the server has not confirmed yet.
• Held wait: the open request that lets the server tell a device about a change at once. See Realtime and scaling.

Besides time, each benchmark records what the code did: how many SQL statements a write took, how many requests and bytes a push needed. These counts are the same on every run and every machine, which makes them the reliable half of the story.

Run themCopy Anchor Link

You need the repository, Bun and Node 24 (the SQLite store and storage run on node:sqlite, which Bun does not have yet).

1. Install the dependencies, from the repository root:

   Copy Text

   bun install

   You should see Bun list the packages it installed, with no errors.

2. Run the benchmarks that need nothing else:

   Copy Text

   bun run bench

   It takes about a minute and a half. You should see one block per benchmark, like this one:

   Copy Text

   ok   sync > sync: bootstrap, a new device downloads everything 10000 rows from the memory store > a new device, up to date
          mean 39.9 ms over 3 runs
          numbers: rowsReceived=10,000, requests=2, pushes=0, pulls=1, bootstraps=1, ...

   mean is the average time of one run. numbers are the counts. A line that starts with FAIL is a benchmark that threw: its message follows.

3. Read the raw report. The run leaves packages/ferrysync/bench-results/results.json, the same file CI uploads. Pass --out some/folder to bun bench/run.ts (in packages/ferrysync) to write it elsewhere.

4. To run the Postgres benchmarks, start Docker and run:

   Copy Text

   bun run bench:postgres

   It starts the database of packages/ferrysync/bench/docker-compose.yml, runs, and stops it. You should see Docker create a ferry-bench-postgres-1 container, five more blocks of results, and the container removed. To use a Postgres of your own, set FERRY_TEST_POSTGRES_URL first (the variable the test suite reads too) and nothing is started.

The Postgres in the compose file is pinned (postgres:17.6-alpine) and limited to 2 CPUs and 1 GB, on a port (54329) that rarely clashes. Pinning is what makes a number mean the same thing next month, on another machine, and in CI.

The numbersCopy Anchor Link

Measured on a laptop, not on CI: an Apple M4 Max (14 cores, 36 GB) running macOS, Node 26.9 and Bun 1.3.6, with nothing else busy. Postgres ran in Docker Desktop, with the compose file's limits. Your numbers will differ; the proportions between rows are what carry over. Each is the mean of 3 to 5 runs (many more for the sub-millisecond ones) after warm-up runs.

The rows are tasks of about 200 bytes of JSON.

The deviceCopy Anchor Link

Opening the app with the rows already on the device, until sync.ready resolves:

The rows take about 4 MB of memory per 10,000 (42 MB for 100,000). The key-value store reads one key per row (about 100,000 calls to open 100,000 rows), where SQLite reads in 5 statements.

With 10,000 rows on the device:

Device and server togetherCopy Anchor Link

The SQLite and memory stores give the same times here: at this size the cost is the client and the protocol, not the store.

The serverCopy Anchor Link

Writes per second is the number of writes in a run divided by its time. Every row is one thread of one process, so these are the speed of one server instance, not of a fleet.

Read them with care. SQLite in memory never touches a disk, so it is a ceiling, not a promise. The real Postgres pays a network hop and a disk flush for every commit, which is what a production database does.

Sixteen devices at once gain only about 2 times on Postgres (1,060 against 520 for one at a time), and that is the design's ceiling, not the hardware's. Every push takes its sync ids from one counter row, with UPDATE ... SET last_sync_id = last_sync_id + n, and the database holds that row's lock until the push commits. So the part of every push that numbers its changes runs one after another, whatever the number of groups or devices: about 1,000 pushes a second here. The rest of each push overlaps, which is where the gain comes from. PGlite has one connection and cannot overlap at all.

A write costs 7 SQL statements on SQLite and 9 on Postgres; a push of one row costs 11 and 15. The 15 is a device's first push, which also creates its row on the server; a push after that costs 13.

Held wait requests. With 1,000 devices (each its own user) holding a wait open on one instance, the server used about 6.6 MB, about 7 KB per held request. Each wait is opened at the current sync id, so it holds, and then one write that concerns all 1,000 wakes every one of them in about 210 ms (205 to 220 ms over repeated runs). Holding costs about 7 statements per waiter (7,003 in all), and waking costs about 7 more (7,016 in all). The server's docs say an instance holds about 2,000: this is the number behind it, and the 10,000-user load test in the roadmap is what will test it past that.

An earlier version of this page said 24 ms. That number measured the wrong thing. The benchmark opened its waits at a sync id read once, at setup, so from its second run on the log was already past that id: every wait answered at once, nothing was held, and the timed write was only a few promises. The benchmark now reads the sync id at each open, checks that no wait has answered before the write, and times only the write and the answers.

What CI does with themCopy Anchor Link

Every release (a vX.Y.Z tag) runs the benchmarks twice in one job, on one machine: once on the previous release, once on the new one. They are the same benchmark files for both, so a release that adds a benchmark is still measured against the old code. The job then compares them. It runs on releases, not on every pull request, because a run takes minutes of shared CPU and most changes do not move these numbers. It does not hold the release back: publishing is a separate workflow.

1. Push a release tag (v0.2.0), or start it by hand: gh workflow run bench.yml (add -f base=<ref> to compare with something other than the latest release, for example main against a release before you merge a risky change). The Benchmarks workflow starts, with one job for the in-process benchmarks and one for Postgres.
2. Open the run's summary page. You should see a table with a row per benchmark: the base's time, the change's time, the difference, and slower where it passed the threshold. The raw JSON of both runs is attached to the run as an artifact (benchmarks-in-process, benchmarks-postgres).
3. Read the two kinds of finding differently:

When a count changes on purpose (a new index, a leaner request), the run fails once, and the release notes say why. The new count is the base for the next release. The first release has no previous one: it only records the numbers.

GotchasCopy Anchor Link

• A shared CI runner is noisy. Other jobs share its CPU, so the same code can differ by 20% or more between runs. That is why a time never fails a build, and why the base runs beside the change instead of being a number written down once. Treat a flag as a reason to run the benchmark again locally.
• The first benchmark of a process is slower. Node compiles code as it runs. The benchmarks warm up first, but compare runs of the same size, not the first against the rest.
• Laptop numbers are not production numbers. Use them to compare changes and storages with each other, not to promise a latency. The load test in the roadmap is for that.
• The SQLite benchmarks use node:sqlite, which Node has and Bun 1.3 does not: the scripts run vitest under Node, started by Bun. Do not run the .bench.ts files with bun directly.
• CI has about 2 vCPUs, and Node and Postgres share them. Postgres numbers from CI are slower than the laptop's for that reason, and that is not a regression. Held-wait timings measured on CI before the fix above are invalid, for the reason given there. Compare CI with CI only.
• Memory numbers are approximate. They read the heap after a forced collection, and they are shown but never compared.
• No Docker, no Postgres numbers. bun run bench skips the Postgres benchmarks, and says nothing about it.

Add a benchmarkCopy Anchor Link

Benchmarks live in packages/ferrysync/bench, in files that end in .bench.ts. They use Vitest's benchmark API, the bench function of the test context:

1. Build the real thing with the helpers in bench/support (createStack makes a server on a real store and clients that reach it over the HTTP binding; storageTarget makes a client storage).
2. Call context.bench(name, fn).run(options) with one of the run options in bench/support/run-options.ts.
3. Call report(context, 'deterministic', { ... }) for counts that never change between runs, and report(context, 'info', { ... }) for numbers that do (memory). Only the first kind fails CI.
4. Run bun run bench and check the new rows appear. Benchmarks are not part of the published package or of test coverage.