Blog
EngineeringFor engineers

Moving a live money system off Firestore without anyone noticing

On 1 October 2026 we moved every record from Firestore to Postgres. We did it by putting one interface in front of the database, teaching Postgres to answer exactly as Firestore did, and proving it byte for byte before the switch.

Lodge Internet · 14 November 2026 · 7 min read

On 1 October 2026 we switched on the maintenance message, stopped the backend, and copied every document out of Firestore. When the backend came back, it was reading and writing Postgres. The first thing it ever wrote there was the setting that lifted the maintenance message.

Everything else had happened in the weeks before, so that the day itself could be boring.

The problem

Lodge Internet sells Wi-Fi codes, and every sale is money. The code that sells them had grown up on Firestore, and it depended on Firestore's behaviour in ways nobody had written down.

Some of it was obvious, and some wasn't:

  • a query with no ordering returns documents in id order;
  • a query ordered by a field silently leaves out documents that don't have that field;
  • 1 equals 1.0, but a boolean is never a number;
  • an update key with a dot in it is a path into nested maps, every dot a step;
  • a missing field and a field set to null are different things, and the code relied on it (a hostel with no isActive field is active).

A rewrite of every query into SQL would have changed some of these by accident, and we would have found out from customers. We needed a way to change the database without changing a single answer.

What we did: branch by abstraction

One interface first, on Firestore

Before any Postgres code existed, we put every read and write behind a data layer in backend/app/repos/. Underneath sits a generic document store (store.py) with a short list of operations: get, query, count, add, set, update, delete, a few batch writes, and run_transaction. Named modules per area (hostels.py, plans.py, controllers.py and the rest) are built on top.

This shipped running on Firestore, with no change in behaviour. One setting, DATASTORE, picks the store.

To keep it honest, tests/test_repo_boundary.py reads every source file and fails the build if anything outside the data layer calls Firestore directly. That test exists because a single direct call would have kept reading and writing Firestore after the cutover, and nothing would have looked wrong.

Firestore's rules, written down once

The rules from the list above, and the rest of them, live in plain Python in backend/app/repos/semantics.py: how values compare, which documents a filter matches, the order a query returns, and what each kind of write does to a document.

Two stores apply them: an in-memory store for fast tests, and the Postgres store. A contract suite (tests/test_repo_store_contract.py) runs the same cases against the Firestore emulator, the in-memory store and Postgres. It runs the Postgres cases twice: once with fields in typed columns, and once with every field pushed into the overflow column. If any store disagrees with the emulator, the suite fails.

Postgres that answers like Firestore

Each collection became a table and each document a row. Two columns carry the awkward parts. extra is a JSONB column holding any field without a column of its own, so nothing is lost. absent_fields lists the mapped fields a document didn't have, so "missing" and "null" stay different.

Money columns are unconstrained numeric, so a whole number and the same number written with .0 stay distinct. That sounds pedantic until you notice the API printed them differently.

Queries work in two passes. SQL narrows the rows to a superset of the answer. Where the narrowing is exact and the database can order rows the way Firestore does, SQL also orders and limits. Either way, semantics.py is applied again to what comes back, so the final answer is Firestore's, whatever SQL did.

Transactions with Firestore's shape

Every write runs in a SERIALIZABLE transaction, retried when Postgres reports a serialization failure or a deadlock, up to five attempts. Five is Firestore's own number.

A transaction also keeps Firestore's discipline: reads first, then writes, all applied at the end. Firestore refuses a read after a write inside a transaction, so ours does too:

def get(self, collection: str, doc_id: str) -> Optional[Doc]:
    if self._writes:
        raise ReadAfterWrite(f"read of {collection}/{doc_id} after a write")
    return self._store._get(self._conn, collection, doc_id)

Code that works on one store works on the other. That was the whole point.

How we proved it

Unit tests weren't enough on their own: the suite didn't reach every path we had touched. So we built make diffcheck.

It runs the same list of scenarios twice, against two versions of the backend (by default, the last commit and the working tree), and records each step's result, every call made to the stubbed outside services, and the whole database after each step. Then it compares the two recordings byte for byte. Identical means the change didn't alter behaviour on those paths.

To make that possible, everything that naturally differs between runs is pinned or normalised. Generated ids come from a counter. PYTHONHASHSEED=0 fixes set ordering, so "first match" logic picks the same document both times. A timestamp written during the run becomes <t@N>, where N is the step that first produced it, so a missing or extra write still shows. Encrypted codes are compared by their decrypted value. The run on the new version also reports which changed statements it actually executed.

make diffcheck-postgres runs the same scenarios on the same code, once on Firestore and once on Postgres. Byte-identical output there was the gate for the switch.

What broke

Keeping behaviour identical meant keeping some behaviour we didn't like. Firestore's client reads every dot in an update key as a path step, even a dot inside a stock pool's name. The low-stock marker was keyed by pool name, so a pool with a dot in its name split its marker into nested maps. We carried that faithfully through the port, because changing it mid-migration would have broken the byte-for-byte check, and fixed it afterwards, on 7 October 2026.

The schema, and what we left loose

backend/app/db/schema.py is the single source for every table, and migrations are generated from it. Migration 0001 is frozen with literal lists, so it keeps doing what it did when it was written, and a test checks that the migrated database still matches the schema file.

The copied tables were deliberately not tightened. No foreign keys on old references, no CHECK constraints on old status fields, no new NOT NULL. Each would make some write that works today fail, and on the money path a refused write is a lost payment record.

The new tables carry the rules Firestore couldn't: unique keys, a double-entry ledger whose postings must balance, append-only history, and an audit log. Postgres is also stricter in three places on purpose. One sale per payment reference. One bank transfer counts once. One voucher per pool, and one username per admin.

The day

Migration 0002 gave the copy tool its own staging schema. Each run stores every document exactly as read, at one instant, as typed JSON. The backend's own database role can't even see it, and runs are pruned once used.

The cutover was planned at about fifteen minutes:

  1. Switch on the lockdown, so the website and the bot show the maintenance message.
  2. Stop the backend. That is the real write freeze. The lockdown alone deliberately keeps webhooks and admin open. Paystack retries webhooks, so a payment made in the window is delivered once the backend is back.
  3. Copy every document into staging, then load it into the real tables.
  4. Verify: every document present, identical type for type, and the money totals equal. If not, start the backend again on Firestore. Nothing has switched.
  5. Mark the database live, switch DATASTORE, start the backend, and lift the lockdown.

So "without anyone noticing" is true from the second customers could buy again, not before. During the window, people saw a maintenance message.

The way back

Firestore was frozen at the cutover and kept as the way back. In Postgres, a trigger records every changed row in a queue. A copy-back tool writes only those documents back to Firestore, and refuses to run unless Postgres is frozen first. We rehearsed the whole thing locally on a pseudonymised copy of production: cutover, a burst of real admin and payment writes, rollback, and a final check that both databases held identical documents.

Firestore was fully retired by 9 November 2026.

The trade-off

Applying Firestore's rules in Python to every query result is slower than trusting SQL, and queries fetch supersets. We took that cost to make the switch safe, then started moving work into SQL where it matters. Stock counts moved into SQL on 7 October 2026.

The schema still carries Firestore's shape. Rows name their hostel rather than pointing at it, so a rename used to mean rewriting every one of them. Migration 0003, dated the day of the cutover, added a real hostel_id link beside each name, and the Postgres store fills it in from the name as each row is written. The name stays, as what the hostel was called at the time.

What's still open

Much of the test suite still runs against the Firestore emulator, and the Firestore store stays in the codebase to serve it. Moving those tests to Postgres, and then deleting the Firestore code and its libraries, is planned and not done. Tightening the loose constraints is a list of counted changes for later.

If you are about to do the same, build the comparison tool before you build the new store. It is what let us say "identical" instead of "it seems fine".

Last checked against the product on 14 November 2026.

More in Engineering