Metrics

Metrics are numbers your server reports about itself, so you can see how it behaves in production without reading logs: how long pushes take, how many devices are connected, how big the log is, why writes are refused. You plug them into whatever monitoring you already run (Prometheus, Grafana, Datadog, OpenTelemetry). Ferry measures; you decide where the numbers go.

Why you would want themCopy Anchor Link

A sync server can look healthy while it is slowly getting worse: pushes taking twice as long, devices falling back to polling, a log growing without bound. Nothing fails, so nothing is logged. A number that you can graph and alert on is how you notice before your users do.

Ferry answers five questions:

1. How long do pushes and pulls take?
2. How many devices is the server holding a connection open for, and how do those end?
3. How big is the log?
4. Why are writes refused?
5. What is retention removing?

Metrics are off by default. Without the metrics option, the server measures nothing and reads no clock.

What a number isCopy Anchor Link

You will meet three kinds:

• A counter only goes up: requests served, writes refused. You graph how fast it grows (a rate).
• A gauge is a value right now: connections open, entries in the log. You graph it as it is.
• A histogram records how long things took, in buckets, so you can ask "how long did the slowest 5% take?" (a percentile).

A tool such as Prometheus keeps these numbers over time. It scrapes your server: every few seconds it fetches a page, here /metrics, and stores what it reads. Grafana draws graphs from that, and alerts fire from it. If you use Datadog or OpenTelemetry instead, the idea is the same.

Where each number is takenCopy Anchor Link

Set it upCopy Anchor Link

Ferry never imports a metrics library. You hand it a small object with three functions, counter, histogram and gauge, that knows how to count. That object is the adapter. The one below keeps the numbers in memory and prints them in the Prometheus text format, which Prometheus, Grafana Agent and the OpenTelemetry Collector all read. With prom-client or OpenTelemetry you would write the same three functions over their instruments.

1. Write the adapter once. Save it as prometheus.ts next to your server.

   Copy Text

   import type \{ MetricLabels, Metrics \} from 'ferrysync/server';
   
   interface Family \{
     name: string;
     help: string;
     kind: 'counter' | 'histogram';
     /** Each line of the family, as text, with its current value. */
     values: Map\<string, number>;
   \}
   
   const byName = ([a]: [string, string], [b]: [string, string]) => (a \< b ? -1 : 1);
   
   /** `ferry_waits_total{held="true",outcome="changed"}`, labels sorted, values made safe. */
   const line = (name: string, labels: MetricLabels = \{\}) => \{
     const text = Object.entries(labels)
       .sort(byName)
       .map(([label, value]) => `${label}="${value.replace(/["\\\n]/g, '_')}"`)
       .join(',');
     return text === '' ? name : `${name}{${text}}`;
   \};
   
   const add = (values: Map\<string, number>, key: string, by: number) => \{
     values.set(key, (values.get(key) ?? 0) + by);
   \};
   
   export const createPrometheusMetrics = () => \{
     const families: Family[] = [];
     const gauges: \{
       name: string;
       help: string;
       observe: () => number | undefined | Promise\<number | undefined>;
     \}[] = [];
   
     const metrics: Metrics = \{
       counter: (\{ name, help \}) => \{
         const values = new Map\<string, number>();
         families.push(\{ name, help, kind: 'counter', values \});
         return (value, labels) => add(values, line(name, labels), value);
       \},
       histogram: (\{ name, help, buckets = [0.01, 0.1, 1, 10] \}) => \{
         const values = new Map\<string, number>();
         families.push(\{ name, help, kind: 'histogram', values \});
         return (value, labels) => \{
           add(values, line(`${name}_count`, labels), 1);
           add(values, line(`${name}_sum`, labels), value);
           for (const bound of [...buckets, Infinity]) \{
             const le = bound === Infinity ? '+Inf' : String(bound);
             add(values, line(`${name}_bucket`, \{ ...labels, le \}), value \<= bound ? 1 : 0);
           \}
         \};
       \},
       gauge: (\{ name, help \}, observe) => \{
         gauges.push(\{ name, help, observe \});
       \},
     \};
   
     /** The body of your `/metrics` route. */
     const render = async () => \{
       const lines: string[] = [];
       for (const \{ name, help, observe \} of gauges) \{
         const value = await observe();
         // No value (the store could not be read): say nothing, never a made-up zero.
         if (value !== undefined) \{
           lines.push(`# HELP ${name} ${help}`, `# TYPE ${name} gauge`, `${name} ${value}`);
         \}
       \}
       for (const \{ name, help, kind, values \} of families) \{
         lines.push(`# HELP ${name} ${help}`, `# TYPE ${name} ${kind}`);
         for (const [text, value] of values) \{
           lines.push(`${text} ${value}`);
         \}
       \}
       return `${lines.join('\n')}\n`;
     \};
   
     return \{ metrics, render \};
   \};

   You should see: nothing yet. It is a file that compiles.

2. Give the server the metrics half, and answer /metrics with the render half.

   Copy Text

   import \{ createSyncServer, memoryStore \} from 'ferrysync/server';
   
   import \{ createPrometheusMetrics \} from './prometheus';
   import \{ schema \} from './schema';
   
   const prometheus = createPrometheusMetrics();
   
   export const syncServer = createSyncServer(\{
     schema,
     store: memoryStore(),
     auth: async (request) => (await getSession(request))?.userId ?? null,
     metrics: prometheus.metrics,
   \});
   
   const metricsResponse = async () =>
     new Response(await prometheus.render(), \{
       headers: \{ 'content-type': 'text/plain; version=0.0.4' \},
     \});
   
   /** Every request: `/metrics` shows the numbers, anything else is for sync. */
   export const handle = (request: Request) =>
     new URL(request.url).pathname === '/metrics' ? metricsResponse() : syncServer.fetch(request);

   Serve handle with your runtime (Bun.serve({ fetch: handle }), for one), or move the two pieces into your framework's routes: syncServer.fetch where the sync route is, and metricsResponse on GET /metrics.

   You should see: your server starts as before. Metrics add one function call per request.

3. Look at the numbers. Use the app, or push once from a script, then ask your server:

   Copy Text

   curl localhost:3000/metrics

   You should see: text with ferry_waits_held 0 and the three ferry_log_* gauges (all 0 on an empty database). A counter or histogram has no line until something happens, so after a few requests you also get lines such as ferry_request_duration_seconds_count{endpoint="push",outcome="ok"} 3.

4. Point your monitoring at it. Tell Prometheus (or your agent) to fetch /metrics every 15 or 30 seconds. Keep that route off the public internet: the numbers say how busy you are.

5. Alert on what matters. Three to start with, in PromQL:

   Copy Text

   histogram_quantile(0.95, sum by (le) (rate(ferry_request_duration_seconds_bucket\{endpoint="push"\}[5m])))
   ferry_waits_held
   rate(ferry_rejections_total\{source="http",code="internal"\}[5m]) > 0

   • The first above your budget: pushes are getting slow.
   • The second far below the number of devices you expect online: realtime has fallen back to polling.
   • The third: the server is failing requests.

This adapter is a teaching sketch: one set of buckets for every histogram unless Ferry suggests others, and no protection against a very large number of label values (Ferry's labels are all bounded). A real one is a thin wrapper over prom-client, whose Counter, Histogram and Gauge take the same names, help strings and label names that Ferry passes in the spec.

Every metricCopy Anchor Link

All names start with ferry_ and follow the Prometheus conventions (snake case, _total for counters, seconds). In OpenTelemetry, counter and histogram map to createCounter and createHistogram, and gauge to createObservableGauge with observe as its callback.

What the labels can be. Every set is fixed, so a metrics backend never grows without limit because of Ferry (the one exception, code, is capped):

OptionsCopy Anchor Link

logSampleMs exists so that scraping often does not read your database often. The log's size is last sync id - floor, two single-row reads, and Ferry takes them together at most once per period, whatever the number of scrapes or gauges. If the read fails, the error goes to onError and the three gauges report nothing for that scrape: your backend keeps its last value or shows a gap. Ferry never invents a number.

Test itCopy Anchor Link

memoryMetrics() in ferrysync/testing is an adapter that keeps everything in plain objects, so a test can read what the server measured:

import { createSyncServer, memoryStore } from 'ferrysync/server';
import { memoryMetrics } from 'ferrysync/testing';

import { schema } from './schema';

const recorded = memoryMetrics();
const server = createSyncServer({
  schema,
  store: memoryStore(),
  auth: async () => 'user-1',
  metrics: recorded.metrics,
});

// …a request…
recorded.count('ferry_request_duration_seconds', { endpoint: 'push', outcome: 'ok' });
await recorded.gauge('ferry_log_entries');
await server.close();

GotchasCopy Anchor Link

• Several instances: sum the held waits, take the max of the log. ferry_waits_held counts what this process holds, so add it up across instances. The log is one log in one database, so every instance reports the same size: use max, never sum, or a three-instance deploy will look three times as full.
• timeout is not always a deadline. It also counts a wait ended by a device that went away and one ended by close(). Only evictions are told apart, in ferry_waits_evicted_total. For a wait that was held, dividing changed by timeout still tells you whether realtime is working.
• Latency stops inside the protocol server. It covers the store, your hooks and the notice sent to other devices. It does not cover auth, reading the body or your framework: measure the full HTTP time there. Your afterCommit hooks count only when you did not pass waitUntil: without it they are awaited before the answer, so they are inside the time; with it they run after the answer and are not.
• A held wait is not a latency. There is no histogram of how long waits are held, because holding is their job. Use ferry_waits_held and ferry_waits_total.
• Rejection codes are capped at 32. The engine's own codes are always labels. A code your validate hook invents counts as its own label up to 32 distinct ones per server, and every new one after that is counted under code="other", so a bug that puts an id in a code cannot flood your metrics backend.
• A metric that throws changes nothing. An instrument that throws, when Ferry declares it or when it records into it, is reported to onError (without counting it in ferry_errors_total, or it could loop). The request is answered as if metrics were off.
• Counters reset when the process restarts. That is normal: use rate() or increase(), not the raw value. On serverless, where instances come and go, a pull-based /metrics route sees only the instance that answers; push the numbers from your adapter instead.
• The log is a count, not bytes. ferry_log_entries does not say how much disk the log takes or how old its oldest entry is. Read those from your database.

NextCopy Anchor Link

• Retention: how long the log is kept, and what ferry_retention_pruned_total counts.
• Realtime and scaling: what held waits are and why many of them is good.