---
title: "Thaub qab, rov qab rau lub sijhawm tshwj xeeb thiab xyaum rov qab"
description: "Thaub qab Quire, rov qab rau lub sijhawm xaiv thiab ua pov thawj ntawm kev xyaum rov qab."
image: "https://docs.quirelms.com/og.png"
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.quirelms.com/hmn/llms.txt
> Use this file to discover all available pages before exploring further.

# Thaub qab, rov qab rau lub sijhawm tshwj xeeb thiab xyaum rov qab

<span id="backup-point-in-time-recovery-and-the-restore-drill"></span>

Tus qauv nyob ntu 8 ntawm `docs/architecture/23-ops.md`. Nov yog phau txheej txheem rau Docker Compose product. Sau kom tus uas tsis yog tus sau tseem ua raws tau; yog kauj ruam twg tsis meej, ntaub ntawv no muaj qhov yuam kev.

## Yam tiv thaiv thiab txoj kev tiv thaiv <!--quire:what-is-protected-and-how-->

| Yam | Txoj kev | Qhov chaw |
| --- | --- | --- |
| Database | Khaws WAL tas li, tsis pub dhau 60 vib nas this ib zaug, txij thawj zaug pib | `pgwal` volume |
| Database | Base backup nrog `pg_basebackup`, raws qhov qub ib hnub ib zaug (`backup-scheduler`) | `pgbackup` volume |
| Database | Luam base backup thiab WAL ua ntaub ntawv foob, txhua tsib feeb (`backup-offsite`) | Store cais uas koj teev |
| Files | `files` volume. Luam siv host backup tool, lossis siv object storage muaj version | `files` volume |
| Secrets | `docker/.env`, tshwj xeeb tshaj `QUIRE_MASTER_KEY` (thiab `QUIRE_MASTER_KEY_RETIRED` uas tseem siv), `QUIRE_BACKUP_ENCRYPTION_KEY`, thiab `docker/secrets/audit-signing-key.pem` | Khaws daim luam tawm ntawm host no |
| Search index, cache thiab rendition | Tsis thaub qab; rov tsim dua | |

Lub hom phiaj: rov qab tau rau hauv 60 vib nas this txij muaj teeb meem, thiab rov qhib database 500 GB hauv 60 feeb.

Ob qho yuam kev uas nquag muaj. Database rov qab **tsis muaj nws cov ntaub ntawv** ua rau nplooj tawg. Database rov qab **tsis muaj `QUIRE_MASTER_KEY`** decrypt tsis tau SSO, webhook thiab credential integration; txog thaum hloov key tiav tsis muaj yam daws tsis tau ([key-rotation.md](/hmn/ops/key-rotation/)), qhov no suav key retired thiab. Ob qho yog ib feem ntawm thaub qab.

## Thaub qab ntaub ntawv <!--quire:taking-backups-->

Base backup ntawm tag nrho cluster:

```sh
docker compose -f docker/compose.yaml --profile backup run --rm backup
```

Nws khaws base backup tshiab tshaj `QUIRE_BACKUP_KEEP` (qhov qub 5) thiab rho WAL qub tshaj uas tsis tas siv lawm, kom archive loj tsis muaj ciam. Teem cron lossis systemd timer ntawm host kom khiav txhua hnub:

```cron
15 2 * * * cd /srv/quire && docker compose -f docker/compose.yaml --profile backup run --rm backup >> /var/log/quire-backup.log 2>&1
```

Lossis cia stack teem nws tus kheej: profile `backup` khiav `backup-scheduler`, uas thaub qab txhua `QUIRE_BACKUP_INTERVAL_HOURS` (qhov qub 24), thiab `backup-offsite` uas piav hauv qab no.

```sh
docker compose -f docker/compose.yaml --profile backup up -d
```

## Luam foob mus host sab nraud <!--quire:encrypted-off-host-copies-->

Ob volume nyob tib host nrog database; thaub qab hauv lub tshuab uas poob lawm tsis suav ua thaub qab. `backup-offsite` luam txhua base backup thiab txhua ntu WAL archive mus store cais ntawm storage port, foob ntaub ntawv thiab khaws raws retention:

- **Kev foob.** AES-256-GCM nrog `QUIRE_BACKUP_ENCRYPTION_KEY` (lossis ntaub ntawv teev hauv `QUIRE_BACKUP_ENCRYPTION_KEY_FILE`): 32 byte, tsim ntawm `openssl rand -hex 32`. Txhua ntaub ntawv muaj nonce thiab authentication tag nyias, yog li tsis muaj key nyeem tsis tau thiab kuaj pom txhua qhov hloov. Khaws key nrog `QUIRE_MASTER_KEY`, kom deb ntawm host no thiab backup store. Tsis muaj key ces rov qab tsis tau.
- **Qhov chaw.** `QUIRE_BACKUP_STORAGE_DRIVER` yog `s3`, `azure` lossis `local` (remote disk mount ntawm `QUIRE_BACKUP_STORAGE_ROOT`). Siv storage setting tib yam tab sis muaj prefix `QUIRE_BACKUP_`: `QUIRE_BACKUP_S3_ENDPOINT`, `QUIRE_BACKUP_S3_BUCKET`, `QUIRE_BACKUP_S3_ACCESS_KEY_ID` thiab lwm yam. Siv bucket txawv thiab yog ua tau as khauj txawv ntawm files; yog provider tso cai, credential yuav sau tau tiam sis rho tsis tau.
- **Retention.** Base backup tshiab tshaj `QUIRE_BACKUP_OFFSITE_KEEP` (qhov qub yog `QUIRE_BACKUP_KEEP`, yog tsis teeb ces 7) thiab WAL uas qhov qub tshaj ntawm lawv xav tau; rho set thiab ntu qub ntawm store.
- **Thaum twg.** Txhua `QUIRE_BACKUP_SHIP_INTERVAL_SECONDS` (qhov qub 300). Kev xa idempotent: yam twb khaws lawm hla; base backup suav tias khaws tiav tsuas yog tom qab sau manifest ua kauj ruam kawg.

Khiav tib command tes ua:

```sh
docker compose -f docker/compose.yaml run --rm backup-offsite bun apps/worker/src/backups/main.ts ship
docker compose -f docker/compose.yaml run --rm backup-offsite bun apps/worker/src/backups/main.ts verify
```

Yuav rov qab ntawm host tshiab, nqa ib set rov los ua ntej, ces siv daim directory uas rub tau hloov `pgbackup` volume thiab `wal-archive` uas rub tau hloov `pgwal` hauv cov kauj ruam hauv qab:

```sh
bun apps/worker/src/backups/main.ts fetch base-20260924T021500Z /srv/restore
```

## Rov qab mus rau lub sijhawm xaiv <!--quire:restoring-to-a-point-in-time-->

Siv tom qab poob ntaub ntawv: import yuam kev, rho chav kawm, lossis xav thim contract migration. Qhov no hloov database khiav tiag, yog li xub xyaum hauv drill hauv qab.

1. **Xaiv lub sijhawm**, UTC, ua ntej muaj teeb meem: `2026-09-24 09:30:00+00`. Audit log (`/admin/audit`) feem ntau qhia lub sijhawm.
2. **Nres txhua yam sau ntaub ntawv**: `docker compose -f docker/compose.yaml stop web content worker scheduler collab`
3. **Khaws cluster puas lawm** kom txog thaum xyuas rov qab tiav:
   ```sh
   docker compose -f docker/compose.yaml stop postgres
   docker run --rm -v quire_postgres18-data:/from -v quire_postgres-damaged:/to alpine cp -a /from/. /to/
   ```
4. **Unpack base backup tshiab tshaj ua ntej lub sijhawm xaiv** rau data volume thiab thov recovery rau lub sijhawm ntawd:
   ```sh
   docker run --rm -v quire_pgbackup:/backups:ro -v quire_postgres18-data:/var/lib/postgresql postgres:18-alpine sh -euc '
     base="$(ls -1d /backups/base-* | sort | tail -n 1)"   # or the one before the target
     rm -rf /var/lib/postgresql/18/docker && mkdir -p /var/lib/postgresql/18/docker
     tar -xzf "$base/base.tar.gz" -C /var/lib/postgresql/18/docker
     touch /var/lib/postgresql/18/docker/recovery.signal
     chown -R postgres:postgres /var/lib/postgresql/18/docker && chmod 700 /var/lib/postgresql/18/docker'
   ```
5. **Recover**: pib Postgres ib zaug nrog chaw recovery, los ntawm Compose override kom tsis txhob hloov ntaub ntawv qub:
   ```yaml
   # docker/compose.recover.yaml
   services:
     postgres:
       command: [postgres, -c, "restore_command=cp /var/lib/postgresql/wal-archive/%f %p",
                 -c, "recovery_target_time=2026-09-24 09:30:00+00",
                 -c, recovery_target_action=promote, -c, archive_mode=off,
                 -c, max_connections=200, -c, hba_file=/etc/postgresql/pg_hba.conf]
   ```
   Override hloov tag command, yog li rov sau ob qho setting uas recovery xav tau: `max_connections` yuav tsum tsis qis dua primary (yog tsis li recovery nres nrog “insufficient parameter settings”) thiab mounted `pg_hba.conf`.
   ```sh
   docker compose -f docker/compose.yaml -f docker/compose.recover.yaml up -d postgres
   docker compose -f docker/compose.yaml logs -f postgres   # wait for "database system is ready"
   ```
6. **Kuaj** ua ntej cia neeg nkag: audit chain (`docker compose -f docker/compose.yaml run --rm worker bun tooling/audit-verify/run.ts`) thiab xyuas tias ntaub ntawv ploj rov los.
7. **Rov qab li qub**: `docker compose -f docker/compose.yaml up -d`. Qhov no pib Postgres nrog archiving thiab pib WAL timeline tshiab. Thaub qab tshiab tam sim ntawd.

Lub npe project `quire` yog prefix rau txhua volume; `docker volume ls` qhia npe meej.

## Drill tshuaj xyuas raws caij <!--quire:the-scheduled-verification-drill-->

`backup-offsite` kuj khiav drill txhua `QUIRE_BACKUP_DRILL_INTERVAL_HOURS` (qhov qub 168, ib lim tiam), thiab rov ua ntawm lwm qhov kev khiav tom qab drill poob. Nws rub base backup off-host tshiab tshaj thiab txhua ntu WAL tom qab, decrypt txhua yam (ua pov thawj tias key qhib tau thiab ntaub ntawv tsis hloov), piv txhua ntaub ntawv rau manifest, xyuas archive yog Postgres data directory, thiab xyuas WAL txij backup mus tom ntej tsis muaj qhov khoob. Sau report rau store ua `reports/drill-<time>.json` thiab service log; drill poob qhia ntaub ntawv lossis ntu xub ploj.

## Xyaum rov qab <!--quire:the-restore-drill-->

Thaub qab uas yeej tsis tau restore tsis yog thaub qab tiag. Drill restore base backup tiav tshiab tshaj uas tsim ua ntej lub sijhawm xaiv nrog WAL archive rau scratch Postgres uas cais ntawm database khiav tiag, ces ua pov thawj:

```sh
docker/scripts/restore-drill.sh                                  # to ninety minutes ago
docker/scripts/restore-drill.sh --target "2026-09-24 09:30:00+00"
```

Lub sijhawm xaiv yuav tsum yog UTC raws nraim hom ntawv no. Yuav tsum muaj base backup laus dua nws thiab archived WAL tom qab nws: thaum nruab tshiab, thaub qab ib zaug thiab tos ntu archive tom ntej (tsis pub dhau ib feeb yog muaj write) ua ntej xaiv sijhawm tom qab backup. Drill xav Docker thiab bash ntawm host xwb.

Ib kauj ruam twg poob drill:

1. **Rov qab tau**: scratch cluster replay txog sijhawm xaiv thiab qhib.
2. **Tiav txaus**: piv row count txhua table rau database nyob (`tooling/restore-drill`). Database nyob twb hloov tom qab lub sijhawm xaiv, yog li table muaj cai txawv tau ntawm qhov ntau dua ntawm 500 row lossis ib feem kaum ntawm nws qhov loj, mus tau ob sab (write ua rau nws poob qab; rho tawm ua rau restore muaj ntau dua); partition tsim tom qab sijhawm xaiv tsis suav ua table ploj. Nthuav ciam rau installation siv hnyav nrog `QUIRE_DRILL_MAX_BEHIND` thiab `QUIRE_DRILL_MAX_DRIFT_RATIO`. Table ploj lossis khoob ua rau poob.
3. **Raug**: audit hash chain kuaj dhau ntawm daim luam restore.
4. **Siv tau**: application role nyeem tau raws row-level security.
5. **Sijhawm**: txij pib txog ntsuab, piv rau `QUIRE_DRILL_RTO_SECONDS` (qhov qub 3600).

Drill yeej tsis sau rau database nyob lossis nws volume: backup thiab WAL volume mount nyeem xwb thiab rho scratch cluster thaum xaus, txawm dhau lossis poob.

Teem `QUIRE_DRILL_REPORT` ua chaw ntaub ntawv kom sau JSON report txawm dhau lossis poob, ces teem khiav ntawm Docker host:

```cron
30 3 1 * * cd /srv/quire && QUIRE_DRILL_REPORT=/var/log/quire-drill.json docker/scripts/restore-drill.sh >> /var/log/quire-drill.log 2>&1
```

Khiav txhua hli thiab ua ntej upgrade. Drill poob ces upgrade raug thaiv. Ib zaug txhua peb hlis, kom ib tug uas tsis tau sau runbook no ua point-in-time restore tiag ntawm host seem, siv daim ntawv no xwb.

## Ntaub ntawv <!--quire:files-->

Cov ntaub ntawv local nyob hauv `files` volume. Thaub qab tib lub sijhawm nrog database thiab rov qab ob qho ua ke:

```sh
docker run --rm -v quire_files:/files:ro -v "$PWD":/out alpine tar -czf /out/files-$(date -u +%Y%m%d).tar.gz -C /files .
```

Yog siv object storage, qhib bucket versioning thiab khaws version qub 35 hnub; ces bucket nws tus kheej pab rov qab ntaub ntawv rau lub sijhawm xaiv.

Source: https://docs.quirelms.com/hmn/ops/backup-restore/index.mdx