Ajiyar bayanai, maido da wani lokaci da atisayen maido

Tsarin yana cikin sashen 8 na docs/architecture/23-ops.md. Wannan jagorar aiki ce ta samfurin Docker Compose. An rubuta ta domin wanda bai rubuta ta ba ya bi ta; idan wani mataki bai bayyana ba, wannan kuskure ne a takardar.

Abin da ake karewa, da yadda

Manufofi: samun damar maido da bayanan da suka wuce gazawa da bai wuce daƙiƙa 60 ba, da maido a cikin minti 60 ga ma’ajiya mai girman 500 GB.

Kurakurai biyu sun fi faruwa. Ma’ajiya da aka dawo da ita ba tare da fayilolinta ba tana nuna shafuka karyayyu. Ma’ajiya da aka dawo da ita ba tare da QUIRE_MASTER_KEY ba ba za ta iya warware bayanan shiga na SSO, webhook da haɗin kai da take riƙe da su ba; har sai an kammala juya maɓallin babba ba tare da abin da ba a warware ba (key-rotation.md), wannan ya haɗa da tsoffin maɓallan ritaya. Duka biyun ɓangare ne na ajiyar bayanai.

Ɗaukar ajiyar bayanai

Ajiyar tushe ta dukan gungu:

docker compose -f docker/compose.yaml --profile backup run --rm backup

Yana riƙe sabbin ajiyoyin tushe QUIRE_BACKUP_KEEP (tsoho 5) kuma yana cire WAL ɗin da mafi tsohuwar ajiyar ba ta ƙara buƙata, domin taskar ba ta girma ba har abada. Saita shi kullum da cron ko agogon systemd a uwar garke:

15 2 * * * cd /srv/quire && docker compose -f docker/compose.yaml --profile backup run --rm backup >> /var/log/quire-backup.log 2>&1

Ko ka bar gungun ya tsara su: bayanin backup yana gudanar da backup-scheduler, wanda ke ɗaukar ajiyar tushe duk bayan QUIRE_BACKUP_INTERVAL_HOURS (tsoho 24), da backup-offsite, wanda aka bayyana gaba.

docker compose -f docker/compose.yaml --profile backup up -d

Kwafi ɓoyayye a wajen uwar garke

Dukkan volume biyun suna kan uwar garke ɗaya da ma’ajiya, kuma ajiyar da ke kan na’urar da ta lalace ba ajiyar bayanai ba ce. backup-offsite yana kwafe kowace ajiyar tushe da kowane ɓangaren WAL da aka adana zuwa wani wurin daban ta tashar ajiya, a ɓoye, ya kuma riƙe su bisa tsawon adanawa:

• Boye bayanai. AES-256-GCM tare da QUIRE_BACKUP_ENCRYPTION_KEY (ko fayil da QUIRE_BACKUP_ENCRYPTION_KEY_FILE ya nuna): bytes 32, daga openssl rand -hex 32. Kowane fayil yana da nonce da alamar tantancewa tasa, saboda haka ba za a iya karanta kwafi ba tare da maɓalli ba, kuma ana gano duk wani canjinsa. Riƙe maɓalli tare da QUIRE_MASTER_KEY, a nesa da wannan uwar garke da wurin ajiyar kwafi. Idan babu maɓalli, babu maido da bayanai.
• Inda. QUIRE_BACKUP_STORAGE_DRIVER yana zama s3, azure ko local (faifan nesa da aka haɗa a QUIRE_BACKUP_STORAGE_ROOT). Saitunan su ne irin na ajiyar fayil da aka ƙara musu QUIRE_BACKUP_: QUIRE_BACKUP_S3_ENDPOINT, QUIRE_BACKUP_S3_BUCKET, QUIRE_BACKUP_S3_ACCESS_KEY_ID da sauransu. Yi amfani da wani bokiti dabam, kuma zai fi kyau wani asusu dabam da na fayiloli, tare da takardun shiga da za su iya rubutawa amma ba sharewa ba idan mai samarwa ya yarda da hakan.
• Tsawon adanawa. Sabbin ajiyoyin tushe QUIRE_BACKUP_OFFSITE_KEEP (tsoho QUIRE_BACKUP_KEEP, in ba haka ba 7) da WAL da mafi tsohuwarsu ke buƙata; ana share tsofaffin saituka da sassa daga wurin ajiyar.
• Lokaci. Duk bayan QUIRE_BACKUP_SHIP_INTERVAL_SECONDS (tsoho 300). Aika ba ya yin abu sau biyu: ana tsallake abin da yake wurin tuni, kuma ana ɗaukar ajiyar tushe ta isa ne idan an rubuta takardar bayananta a ƙarshe.

Ana iya gudanar da waɗannan umarni da hannu:

docker compose -f docker/compose.yaml run --rm backup-offsite bun apps/worker/src/backups/main.ts ship
docker compose -f docker/compose.yaml run --rm backup-offsite bun apps/worker/src/backups/main.ts verify

Domin maido da bayanai a sabon uwar garke, da farko dawo da saitin kwafi, sannan bi matakan da ke ƙasa ta amfani da babban fayil ɗin da aka dawo da shi a maimakon volume na pgbackup, da wal-archive da aka samo a maimakon pgwal:

bun apps/worker/src/backups/main.ts fetch base-20260924T021500Z /srv/restore

Maido zuwa wani lokaci

Yi haka bayan asarar bayanai: shigo da bayanai mara kyau, share kwas, ko ƙaura ta takaita da kake buƙatar soke wa. Yana maye gurbin ma’ajiya mai aiki, don haka ka fara gwada shi da atisayen da ke ƙasa.

1. Zaɓi lokacin manufa, a UTC, kafin lalacewar: 2026-09-24 09:30:00+00. Yawanci rajistar tantancewa (/admin/audit) tana nuna lokacin.
2. Dakatar da duk abin da ke rubutawa: docker compose -f docker/compose.yaml stop web content worker scheduler collab
3. Riƙe gungun da ya lalace har sai an tabbatar da maido:

   docker compose -f docker/compose.yaml stop postgres
   docker run --rm -v quire_postgres18-data:/from -v quire_postgres-damaged:/to alpine cp -a /from/. /to/

4. Buɗe sabuwar ajiyar tushe da ta gabaci lokacin manufa cikin volume ɗin bayanai, sannan ka nemi maido da aka niyya:

   docker run --rm -v quire_pgbackup:/backups:ro -v quire_postgres18-data:/var/lib/postgresql postgres:18-alpine sh -euc '
     base="$(ls -1d /backups/base-* | sort | tail -n 1)"   # or the one before the target
     rm -rf /var/lib/postgresql/18/docker && mkdir -p /var/lib/postgresql/18/docker
     tar -xzf "$base/base.tar.gz" -C /var/lib/postgresql/18/docker
     touch /var/lib/postgresql/18/docker/recovery.signal
     chown -R postgres:postgres /var/lib/postgresql/18/docker && chmod 700 /var/lib/postgresql/18/docker'

5. Mayar da bayanai: kunna Postgres sau ɗaya da saitunan maidowa, daga Compose override domin kada a taɓa babban fayil na yau da kullum:

   # docker/compose.recover.yaml
   services:
     postgres:
       command: [postgres, -c, "restore_command=cp /var/lib/postgresql/wal-archive/%f %p",
                 -c, "recovery_target_time=2026-09-24 09:30:00+00",
                 -c, recovery_target_action=promote, -c, archive_mode=off,
                 -c, max_connections=200, -c, hba_file=/etc/postgresql/pg_hba.conf]

   Override yana maye gurbin dukan umarni, don haka yana maimaita saituna biyu da maidowa ke buƙata: max_connections kada ya yi ƙasa da na uwar garken farko (idan ba haka ba maidowa yana ƙarewa da “insufficient parameter settings”), da pg_hba.conf da aka haɗa.

   docker compose -f docker/compose.yaml -f docker/compose.recover.yaml up -d postgres
   docker compose -f docker/compose.yaml logs -f postgres   # wait for "database system is ready"

6. Duba shi kafin a bari kowa ya shiga: sarkar tantancewa (docker compose -f docker/compose.yaml run --rm worker bun tooling/audit-verify/run.ts), da kuma tabbacin cewa bayanan da suka ɓace sun dawo.
7. Koma yadda aka saba: docker compose -f docker/compose.yaml up -d. Wannan yana sake fara Postgres tare da adana WAL, sabon jerin WAL kuma zai fara. Ɗauki sabuwar ajiyar tushe nan take.

Sunan aikin quire yana zuwa a farkon sunan kowane volume; docker volume ls yana nuna ainihin sunayen.

Atisayen tantancewa da aka tsara

backup-offsite yana gudanar da atisaye duk bayan QUIRE_BACKUP_DRILL_INTERVAL_HOURS (tsoho 168, mako-mako), kuma yana sake yi a zagaye na gaba bayan gazawa. Yana samo sabuwar ajiyar tushe daga wajen uwar garke da kowane ɓangaren WAL bayanta, yana warware kullewar kowane (wanda ke tabbatar maɓalli yana buɗe su kuma babu abin da aka sauya), yana kwatanta fayil da bayanin sa, yana tabbatar taskar kundin bayanai ce ta Postgres, sannan yana duba WAL daga ajiyar gaba babu gibi. Ana rubuta rahoto a wurin ajiyar a matsayin reports/drill-<time>.json da kuma log na hidima; atisayen da ya gaza yana bayyana fayil ko sashin farko da ya ɓace.

Atisayen maido

Ajiyar bayanai da ba a taɓa maidowa ba ba ajiyar bayanai ba ce. Atisayen yana maido da cikakkiyar ajiyar tushe ta ƙarshe kafin manufa tare da taskar WAL, zuwa Postgres na wucin gadi da ba ya raba komai da mai aiki, sannan ya tabbatar da sakamako:

docker/scripts/restore-drill.sh                                  # to ninety minutes ago
docker/scripts/restore-drill.sh --target "2026-09-24 09:30:00+00"

Lokacin manufa dole ne ya zama UTC a daidai wannan tsarin. Ana buƙatar ajiyar tushe da ta girmi sa da WAL da aka adana bayansa: a sabuwar shigarwa, ɗauki ajiyar tushe ka jira sashin da za a adana na gaba (aƙalla minti ɗaya idan ana rubutu) kafin zaɓar manufa bayan ajiyar. Atisaye yana buƙatar Docker da bash a uwar garke kawai.

Kowane mataki yana sa atisayen ya gaza:

1. Iya maidowa: gungun wucin gadi yana kunna WAL har zuwa lokacin manufa, sannan ya buɗe.
2. Cikawa: ƙididdigar layukan kowane tebur idan aka kwatanta da ma’ajiya mai aiki (tooling/restore-drill). Ma’ajiya mai aiki ta ci gaba tun lokacin manufa, saboda haka tebur na iya bambanta da mafi girman adadin layuka 500 ko kashi ɗaya bisa goma na girmansa, ta kowace alkibla (rubutu yana sa maido ya ragu, sharewa kuma ya sa ya fi yawa); partition da aka ƙirƙira bayan lokacin ba tebur da ya ɓace ba ne. Faɗaɗa izinin a shigarwa mai yawan aiki ta QUIRE_DRILL_MAX_BEHIND da QUIRE_DRILL_MAX_DRIFT_RATIO. Tebur da ya ɓace ko ya zama fanko yana sa ya gaza.
3. Mutunci: an tabbatar da sarkar hash ta tantancewa a kwafin da aka dawo.
4. Amfani: matsayin manhaja yana karantawa ta tsaron matakin layi.
5. Lokaci: daga farawa zuwa nasara, bisa QUIRE_DRILL_RTO_SECONDS (tsoho 3600).

Ba ya rubutawa a ma’ajiya ko volume mai aiki: an haɗa volume na ajiyar bayanai da WAL a yanayin karatu kawai, kuma ana cire gungun wucin gadi a ƙarshe, ko ya wuce ko ya gaza.

Saita QUIRE_DRILL_REPORT zuwa hanya domin rubuta rahoton JSON, ko ya wuce ko ya gaza, sannan gudanar da shi bisa jadawali daga Docker host:

30 3 1 * * cd /srv/quire && QUIRE_DRILL_REPORT=/var/log/quire-drill.json docker/scripts/restore-drill.sh >> /var/log/quire-drill.log 2>&1

Gudanar da shi kowane wata da kafin kowane sabuntawa. Atisayen da ya gaza yana hana sabuntawa. Sau ɗaya a kwata, sa wani wanda bai rubuta wannan jagorar ba ya yi maido na gaske zuwa wani lokaci a uwar garke na gefe, ta amfani da wannan takarda kawai.

Fayiloli

Fayilolin gida suna cikin volume na files. Yi musu ajiyar bayanai tare da ma’ajiya, a lokaci guda, sannan a maidowa da su gaba ɗaya:

docker run --rm -v quire_files:/files:ro -v "$PWD":/out alpine tar -czf /out/files-$(date -u +%Y%m%d).tar.gz -C /files .

A ma’ajiyar abubuwa, kunna sigogin bokiti kuma riƙe tsofaffin sigogi na kwana 35; maidowa zuwa wani lokaci ga fayiloli aikin bokiti ne da kansa.