fix: adding db recovery practices (bye bye db)

docs/production-db-recovery-runbook.md

@@ -0,0 +1,197 @@
+# Production DB Recovery and Safety Runbook
+
+Last updated: March 2, 2026
+
+## Purpose
+
+Use this runbook when production data appears lost, reset, or unexpectedly empty.
+
+## Safety Rules (Do Not Bypass)
+
+1. Never run destructive Prisma commands in production:
+- `prisma migrate reset`
+- `prisma migrate dev`
+- `prisma db push --accept-data-loss`
+2. Never run `docker compose down -v` (or `docker-compose down -v`) against production.
+3. Always restore into an isolated database first.
+4. Always take a fresh backup before migration/cutover actions.
+
+## 1) Determine recoverability
+
+### 1.1 Identify active Postgres container + volume
+
+```bash
+docker ps --format '{{.Names}}'
+docker inspect <postgres-container> --format '{{json .Mounts}}'
+```
+
+Record:
+- container name
+- mounted volume name
+- data mount path
+
+### 1.2 Enumerate candidate old volumes
+
+```bash
+docker volume ls | grep -E 'pgdata|skymoney|postgres'
+```
+
+### 1.3 Inspect candidate volumes for PostgreSQL cluster files
+
+```bash
+for v in $(docker volume ls --format '{{.Name}}' | grep -E 'pgdata|skymoney|postgres'); do
+  echo "== $v =="
+  docker run --rm -v "${v}:/var/lib/postgresql/data" alpine sh -lc \
+    "ls -la /var/lib/postgresql/data | head -n 30"
+done
+```
+
+Look for:
+- `PG_VERSION`
+- `base/`
+- `global/`
+- `pg_wal/`
+
+### 1.4 Check filesystem backups and automation history
+
+```bash
+ls -la /opt/skymoney/backups
+ls -la /var/backups
+systemctl list-timers --all | grep -Ei 'backup|postgres|skymoney'
+grep -R \"backup.sh\" /etc/cron* /opt/skymoney 2>/dev/null
+```
+
+Decision:
+- If valid dump/volume exists -> continue to Section 2.
+- If none exists -> mark irrecoverable and continue to Section 3.
+
+## 2) Recover from artifact
+
+### 2.1 Restore into isolated recovery DB
+
+```bash
+export RECOVERY_DB="skymoney_recovery_$(date +%Y%m%d%H%M)"
+export BACKUP_FILE="/opt/skymoney/backups/<chosen-file>.dump"
+export DATABASE_URL="postgres://<admin-user>:<admin-pass>@127.0.0.1:5432/skymoney"
+export RESTORE_DATABASE_URL="postgres://<admin-user>:<admin-pass>@127.0.0.1:5432/${RECOVERY_DB}"
+export RESTORE_DB="$RECOVERY_DB"
+
+cd /opt/skymoney
+./scripts/restore.sh
+```
+
+### 2.2 Validate restored data
+
+```bash
+psql "$RESTORE_DATABASE_URL" -c 'SELECT COUNT(*) FROM "User";'
+psql "$RESTORE_DATABASE_URL" -c 'SELECT COUNT(*) FROM "Transaction";'
+psql "$RESTORE_DATABASE_URL" -c 'SELECT COUNT(*) FROM "EmailToken";'
+```
+
+### 2.3 Cutover (if recovery DB is valid)
+
+1. Put app in brief maintenance mode (or stop writes).
+2. Take a fresh backup of current prod DB.
+3. Restore/copy recovered data into production DB.
+4. Run:
+```bash
+docker-compose exec -T api npx prisma migrate deploy
+```
+5. Remove maintenance mode.
+
+### 2.4 Post-recovery validation
+
+1. Login flow works.
+2. Dashboard and critical routes return expected results.
+3. Security events/logging continue to emit.
+
+## 3) Re-create admin/operator access (if no recovery)
+
+Note: current schema does not include a built-in `isAdmin` role field. This step restores operational access user credentials only.
+
+### 3.1 Create verified operator user
+
+```bash
+cd /opt/skymoney/api
+node -e '
+const { PrismaClient } = require("@prisma/client");
+const argon2 = require("argon2");
+(async () => {
+  const prisma = new PrismaClient();
+  const email = process.env.BOOTSTRAP_EMAIL;
+  const password = process.env.BOOTSTRAP_PASSWORD;
+  if (!email || !password) throw new Error("BOOTSTRAP_EMAIL and BOOTSTRAP_PASSWORD are required");
+  const passwordHash = await argon2.hash(password);
+  await prisma.user.upsert({
+    where: { email },
+    update: { passwordHash, emailVerified: true },
+    create: { email, passwordHash, emailVerified: true },
+  });
+  await prisma.$disconnect();
+  console.log("Bootstrap user ready:", email);
+})().catch((e) => { console.error(e); process.exit(1); });
+'
+```
+
+### 3.2 Credential handling
+
+1. Generate random password in terminal.
+2. Store in password manager.
+3. Rotate immediately after first successful login.
+
+## 4) Backup/restore validation against live VPS
+
+### 4.1 Run real backup
+
+```bash
+cd /opt/skymoney
+BACKUP_ENFORCE_TARGET_CHECK=1 \
+EXPECTED_PROD_DB_HOST=postgres \
+EXPECTED_PROD_DB_NAME=skymoney \
+BACKUP_DIR=/opt/skymoney/backups \
+./scripts/backup.sh
+```
+
+### 4.2 Verify latest checksum
+
+```bash
+LATEST_DUMP="$(ls -1t /opt/skymoney/backups/*.dump | head -n 1)"
+sha256sum -c "${LATEST_DUMP}.sha256"
+```
+
+### 4.3 Restore drill into non-prod DB
+
+```bash
+RESTORE_DB="skymoney_restore_test_$(date +%Y%m%d%H%M)"
+RESTORE_DATABASE_URL="postgres://<admin-user>:<admin-pass>@127.0.0.1:5432/${RESTORE_DB}" \
+DATABASE_URL="postgres://<admin-user>:<admin-pass>@127.0.0.1:5432/skymoney" \
+BACKUP_FILE="$LATEST_DUMP" \
+RESTORE_DB="$RESTORE_DB" \
+./scripts/restore.sh
+```
+
+### 4.4 Validate restore drill
+
+```bash
+psql "postgres://<admin-user>:<admin-pass>@127.0.0.1:5432/${RESTORE_DB}" -c 'SELECT COUNT(*) FROM "User";'
+```
+
+### 4.5 Cleanup drill DB
+
+```bash
+psql "postgres://<admin-user>:<admin-pass>@127.0.0.1:5432/skymoney" \
+  -c "DROP DATABASE IF EXISTS \"${RESTORE_DB}\";"
+```
+
+## 5) Deploy hardening controls
+
+1. `docker-compose.yml` pins volume name: `skymoney_pgdata`.
+2. Deploy workflow sets `COMPOSE_PROJECT_NAME=skymoney`.
+3. Deploy workflow runs `scripts/validate-prod-db-target.sh`.
+4. Deploy workflow runs pre-migration `scripts/backup.sh`.
+5. Deploy workflow uses `prisma migrate deploy` only.
+
+## Quarterly drill requirement
+
+Run a full backup + restore drill every quarter and record evidence in:
+- `tests-results-for-OWASP/evidence-log-template.md`