Backups

RestoringCopy Anchor Link

1. Stop serving sync: take the route down, or close() every instance. Devices retry, and keep working offline meanwhile.

2. Note the highest sync id the lost database handed out. If it still opens, read it there (the table prefix is ferry_ by default):

   Copy Text

   SELECT last_sync_id FROM ferry_state;

   If it does not, take it from your logs, or pick a generous number above it: skipping a billion ids costs nothing.

3. Restore the backup: the sync tables and your own tables together, from the same moment.

4. Run afterRestore once, against the restored database, before any device reaches it:

   Copy Text

   import \{ syncServer \} from './sync-server';
   
   const \{ lastSyncId \} = await syncServer.afterRestore(\{ highestSyncId: 48_210 \});
   console.log(`Sync resumes after sync id ${lastSyncId}`);

5. Delete again the users deleted since the backup, with deleteUser: the backup still holds their data.

6. Redo what must not be lost from the time between the backup and the failure, if you know it: membership removals first (a user removed from a space then is a member again), then additions, then your backend's writes.

7. Serve again.

What devices doCopy Anchor Link

On its next sync, every device downloads its data again, as one that was offline longer than the retention does. Nothing to do in the app, and nothing to tell the user beyond what a full download takes:

• Changes not sent yet are kept and sent afterwards. So are changes whose answer was lost on the way back.
• A device whose earlier changes the server lost sends its pending ones under a new client id, by itself: the server sees that the device's changes start past a gap. A device the backup did not know at all is answered as a forgotten one: it confirms what the restored data shows, and sends the rest again.
• A change the restored database processed but no longer has the answer to (its retention forgot it before the backup) is settled from the rows the device downloads, as without a restore: confirmed only if they show it.
• Rows from after the backup disappear from devices: the database no longer has them.

What is lostCopy Anchor Link

Everything the database had committed after the backup was taken:

Point-in-time recovery shrinks all of these to the last few seconds before the failure.

Why afterRestoreCopy Anchor Link

Devices that synced after the backup hold positions in the log that the restored database has not reached yet. As it takes new changes, it hands out those same positions again, to other changes. A device asks for the changes after its own position, so it would skip the new ones in between and keep rows the server no longer has, without an error anywhere. A change it sent could even count as confirmed because another change took its position.

afterRestore moves the database past every position the lost database handed out, and drops the log of changes (not the rows). Every device's position is then behind the log, and the server tells each one to download again.

GotchasCopy Anchor Link

• Run it before the first request. A device that syncs with the restored database before it may already have skipped changes. Run it even if you think no device synced after the backup: it costs each device one download.
• Too low a highestSyncId leaves devices whose position is above it at risk of skipping changes. Too high costs nothing, up to JavaScript's largest safe integer (2^53 - 1): afterRestore throws a RangeError for a number it cannot add one to, or a negative one.
• Once per restore. Running it again makes every device download once more, and nothing worse.
• Several instances need no restart: run it from one of them, or from a script with the same store.
• On D1, restore with Time Travel and run afterRestore inside the coordinator (from an admin route of the Durable Object): a script with a d1Store of its own would be a second writer. No restart is needed. The coordinator still remembers the newer database, but the restore rolled back the token it checks every commit against, so its first request after the restore fails with D1ConflictError and writes nothing, and it reads D1 again. If that first request is afterRestore, run it again.
• A device restored from its own backup (a phone from iCloud, a copied app folder) is another case, which Ferry handles by itself: the copy gets a new client id the first time it sends a change the original already sent.