Thaub qab, rov qab rau lub sijhawm tshwj xeeb thiab xyaum rov qab

Tus qauv nyob ntu 8 ntawm docs/architecture/23-ops.md. Nov yog phau txheej txheem rau Docker Compose product. Sau kom tus uas tsis yog tus sau tseem ua raws tau; yog kauj ruam twg tsis meej, ntaub ntawv no muaj qhov yuam kev.

Yam tiv thaiv thiab txoj kev tiv thaiv

Lub hom phiaj: rov qab tau rau hauv 60 vib nas this txij muaj teeb meem, thiab rov qhib database 500 GB hauv 60 feeb.

Ob qho yuam kev uas nquag muaj. Database rov qab tsis muaj nws cov ntaub ntawv ua rau nplooj tawg. Database rov qab tsis muaj QUIRE_MASTER_KEY decrypt tsis tau SSO, webhook thiab credential integration; txog thaum hloov key tiav tsis muaj yam daws tsis tau (key-rotation.md), qhov no suav key retired thiab. Ob qho yog ib feem ntawm thaub qab.

Thaub qab ntaub ntawv

Base backup ntawm tag nrho cluster:

docker compose -f docker/compose.yaml --profile backup run --rm backup

Nws khaws base backup tshiab tshaj QUIRE_BACKUP_KEEP (qhov qub 5) thiab rho WAL qub tshaj uas tsis tas siv lawm, kom archive loj tsis muaj ciam. Teem cron lossis systemd timer ntawm host kom khiav txhua hnub:

15 2 * * * cd /srv/quire && docker compose -f docker/compose.yaml --profile backup run --rm backup >> /var/log/quire-backup.log 2>&1

Lossis cia stack teem nws tus kheej: profile backup khiav backup-scheduler, uas thaub qab txhua QUIRE_BACKUP_INTERVAL_HOURS (qhov qub 24), thiab backup-offsite uas piav hauv qab no.

docker compose -f docker/compose.yaml --profile backup up -d

Luam foob mus host sab nraud

Ob volume nyob tib host nrog database; thaub qab hauv lub tshuab uas poob lawm tsis suav ua thaub qab. backup-offsite luam txhua base backup thiab txhua ntu WAL archive mus store cais ntawm storage port, foob ntaub ntawv thiab khaws raws retention:

• Kev foob. AES-256-GCM nrog QUIRE_BACKUP_ENCRYPTION_KEY (lossis ntaub ntawv teev hauv QUIRE_BACKUP_ENCRYPTION_KEY_FILE): 32 byte, tsim ntawm openssl rand -hex 32. Txhua ntaub ntawv muaj nonce thiab authentication tag nyias, yog li tsis muaj key nyeem tsis tau thiab kuaj pom txhua qhov hloov. Khaws key nrog QUIRE_MASTER_KEY, kom deb ntawm host no thiab backup store. Tsis muaj key ces rov qab tsis tau.
• Qhov chaw. QUIRE_BACKUP_STORAGE_DRIVER yog s3, azure lossis local (remote disk mount ntawm QUIRE_BACKUP_STORAGE_ROOT). Siv storage setting tib yam tab sis muaj prefix QUIRE_BACKUP_: QUIRE_BACKUP_S3_ENDPOINT, QUIRE_BACKUP_S3_BUCKET, QUIRE_BACKUP_S3_ACCESS_KEY_ID thiab lwm yam. Siv bucket txawv thiab yog ua tau as khauj txawv ntawm files; yog provider tso cai, credential yuav sau tau tiam sis rho tsis tau.
• Retention. Base backup tshiab tshaj QUIRE_BACKUP_OFFSITE_KEEP (qhov qub yog QUIRE_BACKUP_KEEP, yog tsis teeb ces 7) thiab WAL uas qhov qub tshaj ntawm lawv xav tau; rho set thiab ntu qub ntawm store.
• Thaum twg. Txhua QUIRE_BACKUP_SHIP_INTERVAL_SECONDS (qhov qub 300). Kev xa idempotent: yam twb khaws lawm hla; base backup suav tias khaws tiav tsuas yog tom qab sau manifest ua kauj ruam kawg.

Khiav tib command tes ua:

docker compose -f docker/compose.yaml run --rm backup-offsite bun apps/worker/src/backups/main.ts ship
docker compose -f docker/compose.yaml run --rm backup-offsite bun apps/worker/src/backups/main.ts verify

Yuav rov qab ntawm host tshiab, nqa ib set rov los ua ntej, ces siv daim directory uas rub tau hloov pgbackup volume thiab wal-archive uas rub tau hloov pgwal hauv cov kauj ruam hauv qab:

bun apps/worker/src/backups/main.ts fetch base-20260924T021500Z /srv/restore

Rov qab mus rau lub sijhawm xaiv

Siv tom qab poob ntaub ntawv: import yuam kev, rho chav kawm, lossis xav thim contract migration. Qhov no hloov database khiav tiag, yog li xub xyaum hauv drill hauv qab.

1. Xaiv lub sijhawm, UTC, ua ntej muaj teeb meem: 2026-09-24 09:30:00+00. Audit log (/admin/audit) feem ntau qhia lub sijhawm.
2. Nres txhua yam sau ntaub ntawv: docker compose -f docker/compose.yaml stop web content worker scheduler collab
3. Khaws cluster puas lawm kom txog thaum xyuas rov qab tiav:

   docker compose -f docker/compose.yaml stop postgres
   docker run --rm -v quire_postgres18-data:/from -v quire_postgres-damaged:/to alpine cp -a /from/. /to/

4. Unpack base backup tshiab tshaj ua ntej lub sijhawm xaiv rau data volume thiab thov recovery rau lub sijhawm ntawd:

   docker run --rm -v quire_pgbackup:/backups:ro -v quire_postgres18-data:/var/lib/postgresql postgres:18-alpine sh -euc '
     base="$(ls -1d /backups/base-* | sort | tail -n 1)"   # or the one before the target
     rm -rf /var/lib/postgresql/18/docker && mkdir -p /var/lib/postgresql/18/docker
     tar -xzf "$base/base.tar.gz" -C /var/lib/postgresql/18/docker
     touch /var/lib/postgresql/18/docker/recovery.signal
     chown -R postgres:postgres /var/lib/postgresql/18/docker && chmod 700 /var/lib/postgresql/18/docker'

5. Recover: pib Postgres ib zaug nrog chaw recovery, los ntawm Compose override kom tsis txhob hloov ntaub ntawv qub:

   # docker/compose.recover.yaml
   services:
     postgres:
       command: [postgres, -c, "restore_command=cp /var/lib/postgresql/wal-archive/%f %p",
                 -c, "recovery_target_time=2026-09-24 09:30:00+00",
                 -c, recovery_target_action=promote, -c, archive_mode=off,
                 -c, max_connections=200, -c, hba_file=/etc/postgresql/pg_hba.conf]

   Override hloov tag command, yog li rov sau ob qho setting uas recovery xav tau: max_connections yuav tsum tsis qis dua primary (yog tsis li recovery nres nrog “insufficient parameter settings”) thiab mounted pg_hba.conf.

   docker compose -f docker/compose.yaml -f docker/compose.recover.yaml up -d postgres
   docker compose -f docker/compose.yaml logs -f postgres   # wait for "database system is ready"

6. Kuaj ua ntej cia neeg nkag: audit chain (docker compose -f docker/compose.yaml run --rm worker bun tooling/audit-verify/run.ts) thiab xyuas tias ntaub ntawv ploj rov los.
7. Rov qab li qub: docker compose -f docker/compose.yaml up -d. Qhov no pib Postgres nrog archiving thiab pib WAL timeline tshiab. Thaub qab tshiab tam sim ntawd.

Lub npe project quire yog prefix rau txhua volume; docker volume ls qhia npe meej.

Drill tshuaj xyuas raws caij

backup-offsite kuj khiav drill txhua QUIRE_BACKUP_DRILL_INTERVAL_HOURS (qhov qub 168, ib lim tiam), thiab rov ua ntawm lwm qhov kev khiav tom qab drill poob. Nws rub base backup off-host tshiab tshaj thiab txhua ntu WAL tom qab, decrypt txhua yam (ua pov thawj tias key qhib tau thiab ntaub ntawv tsis hloov), piv txhua ntaub ntawv rau manifest, xyuas archive yog Postgres data directory, thiab xyuas WAL txij backup mus tom ntej tsis muaj qhov khoob. Sau report rau store ua reports/drill-<time>.json thiab service log; drill poob qhia ntaub ntawv lossis ntu xub ploj.

Xyaum rov qab

Thaub qab uas yeej tsis tau restore tsis yog thaub qab tiag. Drill restore base backup tiav tshiab tshaj uas tsim ua ntej lub sijhawm xaiv nrog WAL archive rau scratch Postgres uas cais ntawm database khiav tiag, ces ua pov thawj:

docker/scripts/restore-drill.sh                                  # to ninety minutes ago
docker/scripts/restore-drill.sh --target "2026-09-24 09:30:00+00"

Lub sijhawm xaiv yuav tsum yog UTC raws nraim hom ntawv no. Yuav tsum muaj base backup laus dua nws thiab archived WAL tom qab nws: thaum nruab tshiab, thaub qab ib zaug thiab tos ntu archive tom ntej (tsis pub dhau ib feeb yog muaj write) ua ntej xaiv sijhawm tom qab backup. Drill xav Docker thiab bash ntawm host xwb.

Ib kauj ruam twg poob drill:

1. Rov qab tau: scratch cluster replay txog sijhawm xaiv thiab qhib.
2. Tiav txaus: piv row count txhua table rau database nyob (tooling/restore-drill). Database nyob twb hloov tom qab lub sijhawm xaiv, yog li table muaj cai txawv tau ntawm qhov ntau dua ntawm 500 row lossis ib feem kaum ntawm nws qhov loj, mus tau ob sab (write ua rau nws poob qab; rho tawm ua rau restore muaj ntau dua); partition tsim tom qab sijhawm xaiv tsis suav ua table ploj. Nthuav ciam rau installation siv hnyav nrog QUIRE_DRILL_MAX_BEHIND thiab QUIRE_DRILL_MAX_DRIFT_RATIO. Table ploj lossis khoob ua rau poob.
3. Raug: audit hash chain kuaj dhau ntawm daim luam restore.
4. Siv tau: application role nyeem tau raws row-level security.
5. Sijhawm: txij pib txog ntsuab, piv rau QUIRE_DRILL_RTO_SECONDS (qhov qub 3600).

Drill yeej tsis sau rau database nyob lossis nws volume: backup thiab WAL volume mount nyeem xwb thiab rho scratch cluster thaum xaus, txawm dhau lossis poob.

Teem QUIRE_DRILL_REPORT ua chaw ntaub ntawv kom sau JSON report txawm dhau lossis poob, ces teem khiav ntawm Docker host:

30 3 1 * * cd /srv/quire && QUIRE_DRILL_REPORT=/var/log/quire-drill.json docker/scripts/restore-drill.sh >> /var/log/quire-drill.log 2>&1

Khiav txhua hli thiab ua ntej upgrade. Drill poob ces upgrade raug thaiv. Ib zaug txhua peb hlis, kom ib tug uas tsis tau sau runbook no ua point-in-time restore tiag ntawm host seem, siv daim ntawv no xwb.

Ntaub ntawv

Cov ntaub ntawv local nyob hauv files volume. Thaub qab tib lub sijhawm nrog database thiab rov qab ob qho ua ke:

docker run --rm -v quire_files:/files:ro -v "$PWD":/out alpine tar -czf /out/files-$(date -u +%Y%m%d).tar.gz -C /files .

Yog siv object storage, qhib bucket versioning thiab khaws version qub 35 hnub; ces bucket nws tus kheej pab rov qab ntaub ntawv rau lub sijhawm xaiv.