Papaʻa, hoʻihoʻi manawa kikoʻī, a me ka hoʻāʻo hoʻihoʻi

Aia ka hoʻolālā ma ka māhele 8 o docs/architecture/23-ops.md. ʻO kēia ke alakaʻi hana no ka huahana Docker Compose. Ua kākau ʻia i hiki i kekahi kanaka ʻaʻole nāna i kākau ke hahai; inā pohihihi kekahi ʻanuʻu, he hewa ia o kēia palapala.

Nā mea i pale ʻia a me ke ʻano

Nā pahuhopu: hoʻihoʻi i kahi kūlana ma loko o 60 kekona ma mua o ka pilikia, a hoʻopau i ka hoʻihoʻi ma loko o 60 minuke no ka waihona 500 GB.

ʻElua hewa maʻamau. Hōʻike ka waihona ʻikepili i hoʻihoʻi ʻia me ka ʻole o nā faila i nā ʻaoʻao haki. ʻAʻole hiki i ka waihona ʻikepili i hoʻihoʻi ʻia me ka ʻole o QUIRE_MASTER_KEY ke wehe i nā palapala hōʻoia SSO, webhook, a hoʻohui; a pau ka hoʻololi kī haku me ka ʻole o kekahi mea i koe (key-rotation.md), pili kēia i nā kī hoʻomaha pū. Māhele lāua o ka papaʻa.

Hana i nā papaʻa

Papaʻa kumu o ke kāʻei holoʻokoʻa:

docker compose -f docker/compose.yaml --profile backup run --rm backup

Mālama ia i nā papaʻa kumu hou loa e like me QUIRE_BACKUP_KEEP (5 ma ke ʻano paʻamau), a wehe i ka WAL i pono ʻole i ka papaʻa kahiko loa, i ulu palena ʻole ʻole ai ka waihona. Hoʻonohonoho i ia mea i kēlā me kēia lā me cron a i ʻole timer systemd ma ka host:

15 2 * * * cd /srv/quire && docker compose -f docker/compose.yaml --profile backup run --rm backup >> /var/log/quire-backup.log 2>&1

A i ʻole e ʻae i ka stack e hoʻonohonoho: holo ka profile backup iā backup-scheduler, nāna e hana i papaʻa kumu i kēlā me kēia QUIRE_BACKUP_INTERVAL_HOURS (24 ma ke ʻano paʻamau), a me backup-offsite e wehewehe ʻia aʻe.

docker compose -f docker/compose.yaml --profile backup up -d

Nā papaʻa hoʻopāʻālua ma waho o ka host

Noho nā volume ʻelua ma ka host like me ka waihona ʻikepili; ʻaʻole he papaʻa ka papaʻa ma ka mīkini i hāʻule. Kope ʻo backup-offsite i kēlā me kēia papaʻa kumu a me kēlā me kēia māhele WAL i waihona kaʻawale ma o ka puka waiho ʻikepili; hoʻopāʻālua, a mālama ma lalo o ka lula mālama:

• Hoʻopāʻālua. AES-256-GCM me QUIRE_BACKUP_ENCRYPTION_KEY (a i ʻole ka faila i kapa ʻia e QUIRE_BACKUP_ENCRYPTION_KEY_FILE): 32 byte mai openssl rand -hex 32. Loaʻa i kēlā me kēia faila kona nonce a me ka hōʻailona hōʻoia ponoʻī, no laila ʻaʻole hiki ke heluhelu i kope me ka ʻole o ke kī a ʻike ʻia kekahi hoʻololi. E mālama i ke kī me QUIRE_MASTER_KEY, ma kahi mamao o kēia host a me ka waihona papaʻa. ʻAʻohe hoʻihoʻi me ka ʻole o ke kī.
• Kahi. ʻO QUIRE_BACKUP_STORAGE_DRIVER kekahi o s3, azure, a i ʻole local (disk mamao i kau ʻia ma QUIRE_BACKUP_STORAGE_ROOT). ʻO nā hoʻonohonoho mālama faila me ka prefix QUIRE_BACKUP_: QUIRE_BACKUP_S3_ENDPOINT, QUIRE_BACKUP_S3_BUCKET, QUIRE_BACKUP_S3_ACCESS_KEY_ID, a pēlā aku. E hoʻohana i bucket ʻokoʻa a, ʻoi aku ka maikaʻi, i moʻokāki ʻokoʻa mai nā faila, me nā palapala hōʻoia hiki ke kākau akā ʻaʻole holoi inā ʻae ka mea hoʻolako.
• Mālama. Nā papaʻa kumu hou loa e like me QUIRE_BACKUP_OFFSITE_KEEP (paʻamau QUIRE_BACKUP_KEEP, a i ʻole 7) a me ka WAL e pono ai ka papaʻa kahiko loa; holoi ʻia nā pūʻulu a me nā māhele kahiko mai ka waihona.
• Ka wā. I kēlā me kēia QUIRE_BACKUP_SHIP_INTERVAL_SECONDS (300 ma ke ʻano paʻamau). Hana hoʻokahi wale nō ka hoʻouna: kāpae ʻia nā mea i mālama ʻia, a helu ʻia kahi papaʻa kumu he mālama ʻia ma hope wale nō o ke kākau ʻia ʻana o kāna hōʻike hope.

Holo lima pū kēia kauoha:

docker compose -f docker/compose.yaml run --rm backup-offsite bun apps/worker/src/backups/main.ts ship
docker compose -f docker/compose.yaml run --rm backup-offsite bun apps/worker/src/backups/main.ts verify

No ka hoʻihoʻi ma host hou, e kiʻi mua i pūʻulu, a laila hahai i nā ʻanuʻu ma lalo me ka waihona i kiʻi ʻia ma kahi o ka volume pgbackup, a me ka wal-archive i kiʻi ʻia ma kahi o pgwal:

bun apps/worker/src/backups/main.ts fetch base-20260924T021500Z /srv/restore

Hoʻihoʻi i kekahi manawa kikoʻī

E hoʻohana ma hope o ka nalowale ʻikepili: hoʻokomo hewa, holoi papa, a i ʻole neʻe pani e pono ai ke kāpae. Hoʻololi kēia i ka waihona ʻikepili ola, no laila e hoʻāʻo mua me ka hana ma lalo.

1. Koho i ka manawa pahuhopu, ma UTC, ma mua koke o ka pōʻino: 2026-09-24 09:30:00+00. Hōʻike maʻamau ka moʻolelo hōʻoia (/admin/audit) i ia manawa.
2. Hoʻōki i nā mea a pau e kākau ana: docker compose -f docker/compose.yaml stop web content worker scheduler collab
3. Mālama i ke kāʻei i pohō a hōʻoia ʻia ka hoʻihoʻi:

   docker compose -f docker/compose.yaml stop postgres
   docker run --rm -v quire_postgres18-data:/from -v quire_postgres-damaged:/to alpine cp -a /from/. /to/

4. Wehe i ka papaʻa kumu hou loa ma mua o ka pahuhopu i loko o ka volume ʻikepili, a noi i ka hoʻihoʻi kikoʻī:

   docker run --rm -v quire_pgbackup:/backups:ro -v quire_postgres18-data:/var/lib/postgresql postgres:18-alpine sh -euc '
     base="$(ls -1d /backups/base-* | sort | tail -n 1)"   # or the one before the target
     rm -rf /var/lib/postgresql/18/docker && mkdir -p /var/lib/postgresql/18/docker
     tar -xzf "$base/base.tar.gz" -C /var/lib/postgresql/18/docker
     touch /var/lib/postgresql/18/docker/recovery.signal
     chown -R postgres:postgres /var/lib/postgresql/18/docker && chmod 700 /var/lib/postgresql/18/docker'

5. Hoʻihoʻi: hoʻomaka hoʻokahi manawa iā Postgres me nā hoʻonohonoho hoʻihoʻi, mai ka pani Compose i ʻole e hoʻopā i ka faila maʻamau:

   # docker/compose.recover.yaml
   services:
     postgres:
       command: [postgres, -c, "restore_command=cp /var/lib/postgresql/wal-archive/%f %p",
                 -c, "recovery_target_time=2026-09-24 09:30:00+00",
                 -c, recovery_target_action=promote, -c, archive_mode=off,
                 -c, max_connections=200, -c, hba_file=/etc/postgresql/pg_hba.conf]

   Hoʻololi ka pani i ke kauoha holoʻokoʻa, no laila hana hou i nā hoʻonohonoho ʻelua e pono ai ka hoʻihoʻi: ʻaʻole emi ʻo max_connections ma mua o ka hoʻonohonoho nui (i ʻole pau ka hoʻihoʻi me ka memo «insufficient parameter settings») a me ka pg_hba.conf i kau ʻia.

   docker compose -f docker/compose.yaml -f docker/compose.recover.yaml up -d postgres
   docker compose -f docker/compose.yaml logs -f postgres   # wait for "database system is ready"

6. Nānā iā ia ma mua o ka ʻae ʻana i kekahi: ke kaulahao hōʻoia (docker compose -f docker/compose.yaml run --rm worker bun tooling/audit-verify/run.ts), a hōʻoia ua hoʻi ka ʻikepili i nalowale.
7. Hoʻi i ka maʻamau: docker compose -f docker/compose.yaml up -d. Hoʻomaka hou ʻo Postgres me ka waihona WAL i hoʻā ʻia, a hoʻomaka ka laina manawa WAL hou. Hana koke i papaʻa kumu hou.

Hoʻokomo ka inoa papahana quire i prefix ma kēlā me kēia volume; hōʻike ʻo docker volume ls i nā inoa pololei.

Ka hoʻāʻo hōʻoia i hoʻonohonoho ʻia

Holo pū ʻo backup-offsite i hoʻāʻo i kēlā me kēia QUIRE_BACKUP_DRILL_INTERVAL_HOURS (168, i kēlā me kēia pule ma ke ʻano paʻamau), a i ka holo aʻe ma hope o kekahi hāʻule. Kiʻi ia i ka papaʻa kumu waho hou loa a me kēlā me kēia māhele WAL ma hope ona, wehe i kēlā me kēia hoʻopāʻālua (hōʻoia kēia e wehe mau ke kī a ʻaʻohe mea i hoʻololi), hoʻohālikelike i kēlā me kēia faila me kona hōʻike hope, nānā he papa kuhikuhi ʻikepili Postgres ka waihona, a hōʻoia ʻaʻohe hakahaka ma ka WAL mai ka papaʻa. Kākau ʻia ka hōʻike ma ka waihona ma ke ʻano he reports/drill-<time>.json a ma ka moʻolelo lawelawe; kapa ka hoʻāʻo hāʻule i ka faila a i ʻole ka māhele mua i nalowale.

Ka hoʻāʻo hoʻihoʻi

ʻAʻole he papaʻa ka papaʻa i hoʻihoʻi ʻole ʻia. Hoʻihoʻi ka hoʻāʻo i ka papaʻa kumu piha hou loa ma mua o ka pahuhopu, me ka waihona WAL, i Postgres hoʻāʻo kaʻawale loa mai ka mea ola, a hōʻoia i ka hopena:

docker/scripts/restore-drill.sh                                  # to ninety minutes ago
docker/scripts/restore-drill.sh --target "2026-09-24 09:30:00+00"

Pono ka pahuhopu ma UTC a ma kēlā ʻano kikoʻī. Pono kekahi papaʻa kumu ma mua ona a me WAL i waiho ʻia ma hope: ma kahi hoʻouka hou, e hana i papaʻa kumu a kali i ka māhele waihona aʻe (ʻaʻole i ʻoi aku ma mua o hoʻokahi minuke ke kākau ʻia nā mea), ma mua o ke koho ʻana i pahuhopu ma hope o ka papaʻa. Pono ka hoʻāʻo i Docker a me bash ma ka host; ʻaʻohe mea ʻē aʻe.

Hāʻule ka hoʻāʻo ke hāʻule kekahi ʻanuʻu:

1. Hiki ke hoʻihoʻi: hoʻi ka cluster hoʻāʻo i ka pahuhopu a wehe ʻia.
2. Piha: helu lālani o kēlā me kēia papaʻaina e hoʻohālikelike ʻia me ka waihona ola (tooling/restore-drill). Ua neʻe ka waihona ola ma hope o ka pahuhopu, no laila hiki i kekahi papaʻaina ke ʻokoʻa ma ka waiwai nui aʻe o 500 lālani a i ʻole ka hapaʻumi o kona nui, ma kēlā me kēia ʻaoʻao (hoʻemi nā kākau i nā mea i hoʻihoʻi ʻia; hoʻonui nā holoi ʻana); ʻaʻole helu ʻia kahi māhele i hana ʻia ma hope o ka pahuhopu he papaʻaina nalowale. E hoʻonui i ka ʻae ma kahi hoʻouka hana nui me QUIRE_DRILL_MAX_BEHIND a me QUIRE_DRILL_MAX_DRIFT_RATIO. Hāʻule ka papaʻaina i nele a i hoʻokaʻawale ʻia.
3. Pono: hōʻoia ʻia ke kaulahao hash hōʻoia ma ka kope i hoʻihoʻi ʻia.
4. Hiki ke hoʻohana: heluhelu ke kuleana polokalamu ma lalo o ka palekana pae-lālani.
5. Manawa: mai ka hoʻomaka a hiki i ka holomua, hoʻohālikelike ʻia me QUIRE_DRILL_RTO_SECONDS (3600 ma ke ʻano paʻamau).

ʻAʻole loa ia e kākau i ka waihona ola a me kona mau volume: kau ʻia nā volume papaʻa a WAL ma ke ʻano heluhelu wale nō, a holoi ʻia ke cluster hoʻāʻo i ka hopena, hala a hāʻule paha.

E hoʻonoho iā QUIRE_DRILL_REPORT i helu wahi e kākau ai i hōʻike JSON, inā hala a hāʻule paha; a holo ma ka papa manawa mai ka host Docker:

30 3 1 * * cd /srv/quire && QUIRE_DRILL_REPORT=/var/log/quire-drill.json docker/scripts/restore-drill.sh >> /var/log/quire-drill.log 2>&1

E holo i kēlā me kēia mahina a ma mua o kēlā me kēia hōʻano hou. Kāohi ka hoʻāʻo hāʻule i ka hōʻano hou. Hoʻokahi manawa i ka hapahā makahiki, e noi i kekahi mea ʻaʻole i kākau i kēia alakaʻi e hana i hoʻihoʻi manawa kikoʻī ma host pani me kēia palapala wale nō.

Nā faila

Noho nā faila kūloko ma ka volume files. E hana pū i papaʻa me ka waihona ʻikepili i ka manawa like, a hoʻihoʻi pū iā lāua:

docker run --rm -v quire_files:/files:ro -v "$PWD":/out alpine tar -czf /out/files-$(date -u +%Y%m%d).tar.gz -C /files .

Me ka waihona mea, hoʻā i ka mālama mana bucket a mālama i nā mana kahiko ʻole no 35 lā; na ka bucket ponoʻī ka hoʻihoʻi manawa kikoʻī o nā faila ma ia manawa.