Ndabere, mgbake ruo oge, na mmega iweghachi

Nhazi ahụ dị na docs/architecture/23-ops.md ngalaba 8. Nke a bụ akwụkwọ ọrụ maka ngwaahịa Docker Compose. E dere ya ka onye na-edeghị ya soro ya; ọ bụrụ na nzọụkwụ edoghị anya, nke ahụ bụ ntụpọ n’akwụkwọ a.

Ihe echekwara, na otu

Ebumnobi: ebe mgbake n’ime sekọnd 60 nke ọdịda, na iweghachi n’ime nkeji 60 maka nchekwa data 500 GB.

Njehie abụọ na-emekarị. Nchekwa data eweghachiri na-enweghị faịlụ ya na-egosi peeji gbajiri. Nchekwa data eweghachiri na-enweghị QUIRE_MASTER_KEY enweghị ike iwepu nzuzo na nzere SSO, webhook na njikọta o ji; ruo mgbe ntụgharị igodo ukwu gwụsịrị na-enweghị ihe a na-edozibeghị (key-rotation.md), nke ahụ gụnyere igodo lara ezumike nká. Ha abụọ bụ akụkụ nke ndabere.

Iwere ndabere

Mbipụta ndabere mbụ nke ụyọkọ dum:

docker compose -f docker/compose.yaml --profile backup run --rm backup

Ọ na-edebe mbipụta ndabere mbụ QUIRE_BACKUP_KEEP kachasị ọhụrụ (ndabara 5) wee belata WAL nke kacha ochie na-achọghịzi, ka ebe ndebe ghara itolite na-enweghị oke. Hazie ya kwa ụbọchị na cron ma ọ bụ systemd timer na họst:

15 2 * * * cd /srv/quire && docker compose -f docker/compose.yaml --profile backup run --rm backup >> /var/log/quire-backup.log 2>&1

Ma ọ bụ hapụ stack ka ọ hazie ya: profaịlụ backup na-agba backup-scheduler, nke na-ewere mbipụta ndabere mbụ kwa QUIRE_BACKUP_INTERVAL_HOURS (ndabara 24), na backup-offsite, akọwara na-esote.

docker compose -f docker/compose.yaml --profile backup up -d

Mbipụta nzuzo n’èzí họst

Volum abụọ ahụ bi n’otu họst dịka nchekwa data, ndabere dị na igwe dara abụghị ndabere. backup-offsite na-edetuo mbipụta ndabere mbụ ọ bụla na akụkụ WAL echekwara ọ bụla na ụlọ ahịa dị iche site n’ọdụ ụgbọ nchekwa, na nzuzo, ma debe ha ebe ahụ n’okpuru njide:

• Nzuzo. AES-256-GCM na QUIRE_BACKUP_ENCRYPTION_KEY (ma ọ bụ faịlụ nke QUIRE_BACKUP_ENCRYPTION_KEY_FILE kpọrọ aha): 32 bytes, site na openssl rand -hex 32. Faịlụ ọ bụla nwere nonce nke ya na akara nyocha, ya mere mbipụta anaghị agụ na-enweghị igodo a na-achọpụta mgbanwe ọ bụla na ya. Debe igodo na QUIRE_MASTER_KEY, pụọ na họst a na pụọ na ụlọ ahịa ndabere. Na-enweghị igodo enweghị iweghachi.
• Ebe. QUIRE_BACKUP_STORAGE_DRIVER bụ s3, azure ma ọ bụ local (a disk dịpụrụ adịpụ agbagoro na QUIRE_BACKUP_STORAGE_ROOT). Ntọala bụ nke nchekwa faịlụ nwere prefix QUIRE_BACKUP_: QUIRE_BACKUP_S3_ENDPOINT, QUIRE_BACKUP_S3_BUCKET, QUIRE_BACKUP_S3_ACCESS_KEY_ID, na ndị ọzọ. Jiri bucket dị iche na, ọ kacha mma, akaụntụ dị iche site na faịlụ, na nzere nwere ike ide ma ghara ihichapụ ọ bụrụ na onye na-enye kwere ya.
• Njide. Mbipụta ndabere mbụ QUIRE_BACKUP_OFFSITE_KEEP kachasị ọhụrụ (ndabara QUIRE_BACKUP_KEEP, ma ọ bụ 7) na WAL nke kacha ochie n’ime ha chọrọ; a na-ehichapụ nhazi na akụkụ ochie na ụlọ ahịa.
• Mgbe. Kwa QUIRE_BACKUP_SHIP_INTERVAL_SECONDS (ndabara 300). Mbupu bụ idempotent: a na-awụli ihe echekwara, a na-agụkwa mbipụta ndabere mbụ dịka echekwara naanị ozugbo edere manifest ya, n’ikpeazụ.

Otu iwu ahụ na-agba n’aka:

docker compose -f docker/compose.yaml run --rm backup-offsite bun apps/worker/src/backups/main.ts ship
docker compose -f docker/compose.yaml run --rm backup-offsite bun apps/worker/src/backups/main.ts verify

Iji weghachite na họst ọhụrụ, weta nhazi azụ nke mbụ, emesịa soro nzọụkwụ dị n’okpuru na ndekọ ewetara n’ọnọdụ volum pgbackup na wal-archive ewetara n’ọnọdụ pgwal:

bun apps/worker/src/backups/main.ts fetch base-20260924T021500Z /srv/restore

Iweghachi ruo oge

Jiri nke a mgbe data furu: mbubata ọjọọ, nkuzi ehichapụrụ, mbugharị nkwekọrịta ị chọrọ ịmegharị. Ọ na-anọchi nchekwa data dị ndụ, ya mere jiri mmega dị n’okpuru megharịa ya nke mbụ.

1. Họrọ oge ebumnobi, na UTC, obere oge tupu mmebi: 2026-09-24 09:30:00+00. Log nyocha (/admin/audit) na-egosikarị oge ahụ.
2. Kwụsị ihe niile na-ede: docker compose -f docker/compose.yaml stop web content worker scheduler collab
3. Debe ụyọkọ mebiri ruo mgbe enyochara iweghachi:

   docker compose -f docker/compose.yaml stop postgres
   docker run --rm -v quire_postgres18-data:/from -v quire_postgres-damaged:/to alpine cp -a /from/. /to/

4. Mepee mbipụta ndabere mbụ kachasị ọhụrụ ochie karịa ebumnobi na volum data, ma rịọ mgbake ezubere:

   docker run --rm -v quire_pgbackup:/backups:ro -v quire_postgres18-data:/var/lib/postgresql postgres:18-alpine sh -euc '
     base="$(ls -1d /backups/base-* | sort | tail -n 1)"   # or the one before the target
     rm -rf /var/lib/postgresql/18/docker && mkdir -p /var/lib/postgresql/18/docker
     tar -xzf "$base/base.tar.gz" -C /var/lib/postgresql/18/docker
     touch /var/lib/postgresql/18/docker/recovery.signal
     chown -R postgres:postgres /var/lib/postgresql/18/docker && chmod 700 /var/lib/postgresql/18/docker'

5. Gbakee: malite Postgres otu ugboro na ntọala mgbake, site na Compose override ka faịlụ nkịtị ghara imetụ:

   # docker/compose.recover.yaml
   services:
     postgres:
       command: [postgres, -c, "restore_command=cp /var/lib/postgresql/wal-archive/%f %p",
                 -c, "recovery_target_time=2026-09-24 09:30:00+00",
                 -c, recovery_target_action=promote, -c, archive_mode=off,
                 -c, max_connections=200, -c, hba_file=/etc/postgresql/pg_hba.conf]

   Override na-anọchi iwu dum, ya mere ọ na-emegharị ntọala abụọ ahụ mgbake dabere na ha: max_connections adịghị ala karịa nke primary (ma ọ bụghị mgbake na-akwụsị na “insufficient parameter settings”) na pg_hba.conf agbagoro.

   docker compose -f docker/compose.yaml -f docker/compose.recover.yaml up -d postgres
   docker compose -f docker/compose.yaml logs -f postgres   # wait for "database system is ready"

6. Lelee ya tupu ikwe onye ọ bụla bata: agbụ nyocha (docker compose -f docker/compose.yaml run --rm worker bun tooling/audit-verify/run.ts), na na data furu alọghachila.
7. Laghachi na nkịtị: docker compose -f docker/compose.yaml up -d. Nke a na-amalitegharị Postgres na archiving, usoro WAL ọhụrụ amalite. Were mbipụta ndabere mbụ ọhụrụ ozugbo.

Aha ọrụ quire na-ebu ụzọ na volum ọ bụla; docker volume ls na-egosi aha kpọmkwem.

Mmega nkwenye ahaziri

backup-offsite na-agbakwa mmega kwa QUIRE_BACKUP_DRILL_INTERVAL_HOURS (ndabara 168, kwa izu), na ọzọ na pass na-esote mgbe otu dara. Ọ na-eweta mbipụta ndabere mbụ n’èzí họst kachasị ọhụrụ na akụkụ WAL ọ bụla mgbe ya, na-ewepu nzuzo na nke ọ bụla (nke gosipụtara na igodo ka na-emepe ha na ọ dịghị ihe gbanwere), na-atụnyere nke ọ bụla faịlụ na manifest ya, na-enyocha na ebe ndebe bụ ndekọ data Postgres, na na-enyocha na WAL site na ndabere gaa n’ihu enweghị oghere. A na-ede akụkọ na ụlọ ahịa dịka reports/drill-<time>.json na na log ọrụ; mmega dara na-akpọ faịlụ ma ọ bụ akụkụ mbụ na-efu.

Mmega iweghachi

Ndabere a na-eweghachibeghị abụghị ndabere. Mmega na-eweghachi mbipụta ndabere mbụ zuru oke kachasị ọhụrụ ewere tupu ebumnobi, gbakwunyere ebe ndebe WAL, na Postgres scratch nke na-ekerịtaghị ihe ọ bụla na nke dị ndụ, ma gosipụtara nsonaazụ:

docker/scripts/restore-drill.sh                                  # to ninety minutes ago
docker/scripts/restore-drill.sh --target "2026-09-24 09:30:00+00"

Ebumnobi bụ UTC n’ụdị ahụ kpọmkwem. Ọ chọrọ mbipụta ndabere mbụ ochie karịa ya na WAL echekwara gafere ya: na nrụnye ọhụrụ, were mbipụta ndabere mbụ wee chere akụkụ echekwara na-esote (karịa otu nkeji na ederede) tupu ịhọrọ ebumnobi mgbe ndabere. Mmega chọrọ Docker na bash na họst, ọ dịghị ihe ọzọ.

Nzọụkwụ ọ bụla na-ada mmega:

1. Mgbake: ụyọkọ scratch na-emegharị ruo ebumnobi wee mepee.
2. Zuru oke: ọnụọgụ ahịrị tebụl ọ bụla megide nchekwa data dị ndụ (tooling/restore-drill). Nchekwa data dị ndụ agaala n’ihu kemgbe ebumnobi, ya mere tebụl nwere ike ịdị iche site na nke ka ukwuu nke ahịrị 500 na otu ụzọ n’ụzọ iri nke nha ya, na ụzọ ọ bụla (ederede na-eme ka ọ laa azụ, ihichapụ na-eme ka iweghachi jide karịa); nkebi emepụtara mgbe ebumnobi abụghị tebụl furu. Gbasaa ohere na nrụnye na-arụsi ọrụ ike na QUIRE_DRILL_MAX_BEHIND na QUIRE_DRILL_MAX_DRIFT_RATIO. Tebụl na-efu ma ọ bụ nke tọgbọrọ chakoo na-ada.
3. Iguzosi ike: agbụ hash nyocha na-enyocha na mbipụta eweghachiri.
4. Ojiji: ọrụ ngwa na-agụ site na nchebe ọkwa ahịrị.
5. Oge: mmalite ruo akwụkwọ ndụ, megide QUIRE_DRILL_RTO_SECONDS (ndabara 3600).

Ọ dịghị mgbe ọ na-ede na nchekwa data dị ndụ ma ọ bụ volum ya: volum ndabere na WAL agbagoro naanị ọgụgụ a na-ewepụ ụyọkọ scratch na njedebe, ma ọ gafere ma ọ daa.

Tọọ QUIRE_DRILL_REPORT na ụzọ ka e dee akụkọ JSON, ma ọ gafere ma ọ daa, ma gbaa ya na nhazi site na họst Docker:

30 3 1 * * cd /srv/quire && QUIRE_DRILL_REPORT=/var/log/quire-drill.json docker/scripts/restore-drill.sh >> /var/log/quire-drill.log 2>&1

Gbaa ya kwa ọnwa na tupu mmelite ọ bụla. Mmega dara na-egbochi mmelite. Otu ugboro n’otu ụzọ n’ụzọ anọ, mee ka onye na-edeghị akwụkwọ ọrụ a mee ezigbo iweghachi ruo oge na họst mapụtara, na-eji naanị akwụkwọ a.

Faịlụ

Faịlụ mpaghara bi na volum files. Debe ndabere ya na nchekwa data, na otu oge, ma weghachite ha abụọ ọnụ:

docker run --rm -v quire_files:/files:ro -v "$PWD":/out alpine tar -czf /out/files-$(date -u +%Y%m%d).tar.gz -C /files .

Na nchekwa ihe, gbanye bucket versioning ma debe ụbọchị 35 nke ụdị na-abụghị nke ugbu a; mgbake ruo oge maka faịlụ bụzi nke bucket n’onwe ya.