Serep, pamulihan point-in-time lan latihan restore

Desaine iku docs/architecture/23-ops.md bagean 8. Iki runbook kanggo produk Docker Compose. Ditulis supaya ditindakake dening wong sing ora nulis; yen langkah ora cetha, iku cacat ing dokumen iki.

Apa sing dilindhungi, lan carane

Target: titik pamulihan sajrone 60 detik saka gagal, lan restore sajrone 60 menit kanggo database 500 GB.

Rong kaluputan umum. Database sing dibalekake tanpa file-e ngrender kaca rusak. Database sing dibalekake tanpa QUIRE_MASTER_KEY ora bisa dekripsi kredensial SSO, webhook lan integrasi sing dicekel; nganti rotasi kunci master rampung tanpa sing ora dirampungake (key-rotation.md), iku kalebu kunci sing dipensiunake. Kalorone iku bagean saka serep.

Njupuk serep

Serep dasar kabeh cluster:

docker compose -f docker/compose.yaml --profile backup run --rm backup

Iki njaga serep dasar QUIRE_BACKUP_KEEP paling anyar (gawan 5) lan motong WAL sing ora dibutuhake sing paling lawas maneh, supaya arsip ora bisa tuwuh tanpa wates. Jadwalake saben dina nganggo cron utawa timer systemd ing host:

15 2 * * * cd /srv/quire && docker compose -f docker/compose.yaml --profile backup run --rm backup >> /var/log/quire-backup.log 2>&1

Utawa jarake stack jadwalake: profil backup mbukak backup-scheduler, sing njupuk serep dasar saben QUIRE_BACKUP_INTERVAL_HOURS (gawan 24), lan backup-offsite, sing diterangake sabanjure.

docker compose -f docker/compose.yaml --profile backup up -d

Salinan off-host dienkripsi

Kalorone volume manggon ing host sing padha karo database, lan serep ing mesin sing gagal dudu serep. backup-offsite nyalin saben serep dasar lan saben segmen WAL sing diarsipake menyang toko kapisah liwat port panyimpenan, dienkripsi, lan njaga ing kana ing sangisore retensi:

• Enkripsi. AES-256-GCM nganggo QUIRE_BACKUP_ENCRYPTION_KEY (utawa file sing dijenengi dening QUIRE_BACKUP_ENCRYPTION_KEY_FILE): 32 byte, saka openssl rand -hex 32. Saben file duwe nonce dhewe lan tag autentikasi, supaya salinan ora bisa diwaca tanpa kunci lan owahan apa wae dideteksi. Simpen kunci karo QUIRE_MASTER_KEY, adoh saka host iki lan adoh saka toko serep. Tanpa kunci ora ana restore.
• Ing ngendi. QUIRE_BACKUP_STORAGE_DRIVER iku s3, azure utawa local (disk remote sing dipasang ing QUIRE_BACKUP_STORAGE_ROOT). Setelane iku setelan panyimpenan file kanthi prefiks QUIRE_BACKUP_: QUIRE_BACKUP_S3_ENDPOINT, QUIRE_BACKUP_S3_BUCKET, QUIRE_BACKUP_S3_ACCESS_KEY_ID, lan sapanunggalane. Gunakake bucket beda lan, idealnya, akun beda saka file, kanthi kredensial sing bisa nulis nanging ora mbusak yen panyedhiya ngidinake.
• Retensi. Serep dasar QUIRE_BACKUP_OFFSITE_KEEP paling anyar (gawan QUIRE_BACKUP_KEEP, yen ora 7) lan WAL sing dibutuhake sing paling lawas; set lan segmen luwih lawas dibusak saka toko.
• Kapan. Saben QUIRE_BACKUP_SHIP_INTERVAL_SECONDS (gawan 300). Pangiriman idempoten: apa sing wis disimpen dilewati, lan serep dasar dietung disimpen mung yen manifeste ditulis, pungkasan.

Prentah sing padha mlaku kanthi tangan:

docker compose -f docker/compose.yaml run --rm backup-offsite bun apps/worker/src/backups/main.ts ship
docker compose -f docker/compose.yaml run --rm backup-offsite bun apps/worker/src/backups/main.ts verify

Kanggo mbalekake ing host anyar, gawa set bali dhisik, banjur tindakake langkah ing ngisor kanthi direktori sing dijupuk ngganti volume pgbackup lan wal-archive sing dijupuk ngganti pgwal:

bun apps/worker/src/backups/main.ts fetch base-20260924T021500Z /srv/restore

Mbalekake menyang titik wektu

Gunakake iki sawise data ilang: impor ala, kursus sing dibusak, migrasi kontrak sing perlu dibatalake. Iki ngganti database live, dadi latih dhisik nganggo latihan ing ngisor.

1. Pilih wektu target, ing UTC, sakdurunge rusak: 2026-09-24 09:30:00+00. Log audit (/admin/audit) biasane nuduhake wektune.
2. Mungkasi kabeh sing nulis: docker compose -f docker/compose.yaml stop web content worker scheduler collab
3. Njaga cluster rusak nganti restore diverifikasi:

   docker compose -f docker/compose.yaml stop postgres
   docker run --rm -v quire_postgres18-data:/from -v quire_postgres-damaged:/to alpine cp -a /from/. /to/

4. Bongkar serep dasar paling anyar sing luwih lawas saka target menyang volume data, lan njaluk pamulihan target:

   docker run --rm -v quire_pgbackup:/backups:ro -v quire_postgres18-data:/var/lib/postgresql postgres:18-alpine sh -euc '
     base="$(ls -1d /backups/base-* | sort | tail -n 1)"   # or the one before the target
     rm -rf /var/lib/postgresql/18/docker && mkdir -p /var/lib/postgresql/18/docker
     tar -xzf "$base/base.tar.gz" -C /var/lib/postgresql/18/docker
     touch /var/lib/postgresql/18/docker/recovery.signal
     chown -R postgres:postgres /var/lib/postgresql/18/docker && chmod 700 /var/lib/postgresql/18/docker'

5. Pulihake: miwiti Postgres sapisan kanthi setelan pamulihan, saka override Compose supaya file normal ora didemek:

   # docker/compose.recover.yaml
   services:
     postgres:
       command: [postgres, -c, "restore_command=cp /var/lib/postgresql/wal-archive/%f %p",
                 -c, "recovery_target_time=2026-09-24 09:30:00+00",
                 -c, recovery_target_action=promote, -c, archive_mode=off,
                 -c, max_connections=200, -c, hba_file=/etc/postgresql/pg_hba.conf]

   Override ngganti kabeh prentah, dadi mbaleni rong setelan sing gumantung pamulihan: max_connections ora luwih endhek saka primary (yen ora pamulihan dibatalake kanthi “insufficient parameter settings”) lan pg_hba.conf sing dipasang.

   docker compose -f docker/compose.yaml -f docker/compose.recover.yaml up -d postgres
   docker compose -f docker/compose.yaml logs -f postgres   # wait for "database system is ready"

6. Priksa sadurunge ngidinake sapa wae mlebu: rante audit (docker compose -f docker/compose.yaml run --rm worker bun tooling/audit-verify/run.ts), lan manawa data sing ilang bali.
7. Bali menyang normal: docker compose -f docker/compose.yaml up -d. Iki miwiti maneh Postgres kanthi arsip urip, lan timeline WAL anyar diwiwiti. Jupuk serep dasar anyar langsung.

Jeneng proyek quire ngawali saben volume; docker volume ls nuduhake jeneng persis.

Latihan verifikasi terjadwal

backup-offsite uga mbukak latihan saben QUIRE_BACKUP_DRILL_INTERVAL_HOURS (gawan 168, mingguan), lan maneh ing pass sabanjure sawise gagal siji. Iki njupuk serep dasar off-host paling anyar lan saben segmen WAL sawise iku, dekripsi saben (sing mbuktekake kunci isih mbukak lan ora ana sing diowahi), mbandhingake saben file karo manifeste, mriksa arsip iku direktori data Postgres, lan mriksa WAL saka serep terus ora ana jeda. Laporan ditulis menyang toko minangka reports/drill-<time>.json lan menyang log layanane; latihan gagal nyebutake file utawa segmen ilang kapisan.

Latihan restore

Serep sing ora tau dibalekake dudu serep. Latihan mbalekake serep dasar lengkap paling anyar sing dijupuk sadurunge target, ditambah arsip WAL, menyang Postgres scratch sing ora nuduhake apa-apa karo sing live, lan mbuktekake asile:

docker/scripts/restore-drill.sh                                  # to ninety minutes ago
docker/scripts/restore-drill.sh --target "2026-09-24 09:30:00+00"

Target iku UTC ing wujud persis kasebut. Butuh serep dasar sing luwih lawas saka iku lan WAL sing diarsipake ngluwihi iku: ing instalasi anyar, jupuk serep dasar lan ngenteni segmen sing diarsipake sabanjure (paling suwe semenit kanthi tulisan) sadurunge milih target sawise serep. Latihan butuh Docker lan bash ing host, ora ana liyane.

Saben langkah nggagalake latihan:

1. Bisa dipulihake: cluster scratch muter maneh menyang target lan mbukak.
2. Lengkap: cacah baris saben tabel nglawan database live (tooling/restore-drill). Database live wis maju wiwit target, dadi tabel bisa beda kanthi sing luwih gedhe saka 500 baris lan sepersepuluh ukurane, ing arah endi wae (tulisan nggawe ketinggalan, pambusakan nggawe restore nyekel luwih akeh); partisi sing digawe sawise target dudu tabel ilang. Ambakake kelonggaran ing instalasi luwih rame nganggo QUIRE_DRILL_MAX_BEHIND lan QUIRE_DRILL_MAX_DRIFT_RATIO. Tabel ilang utawa kosong gagal.
3. Integritas: rante hash audit verifikasi ing salinan sing dibalekake.
4. Bisa digunakake: peran aplikasi maca liwat keamanan tingkat baris.
5. Wektu: miwiti nganti ijo, nglawan QUIRE_DRILL_RTO_SECONDS (gawan 3600).

Ora tau nulis menyang database live utawa volume-e: volume serep lan WAL dipasang read-only lan cluster scratch dibusak ing pungkasan, lulus utawa gagal.

Setel QUIRE_DRILL_REPORT menyang path supaya laporan JSON ditulis, lulus utawa gagal, lan mbukak ing jadwal saka host Docker:

30 3 1 * * cd /srv/quire && QUIRE_DRILL_REPORT=/var/log/quire-drill.json docker/scripts/restore-drill.sh >> /var/log/quire-drill.log 2>&1

Mbukak saben sasi lan sadurunge saben upgrade. Latihan gagal mblokir upgrade. Sapisan seperapat, njaluk wong sing ora nulis runbook iki nindakake restore point-in-time nyata ing host serep, nggunakake mung dokumen iki.

File

File lokal manggon ing volume files. Serep bebarengan karo database, ing wektu sing padha, lan balekake kalorone bebarengan:

docker run --rm -v quire_files:/files:ro -v "$PWD":/out alpine tar -czf /out/files-$(date -u +%Y%m%d).tar.gz -C /files .

Kanthi panyimpenan objek, uripake versioning bucket lan njaga 35 dina versi noncurrent; pamulihan point-in-time kanggo file banjur duweke bucket dhewe.