# Backup & Restore System — Deployment Guide (Ubuntu VPS)

This guide covers deploying the Backup & Restore System on an Ubuntu VPS,
including `mysqldump`, cron scheduling, storage, and security hardening.

---

## 1. Prerequisites

Install MySQL client tools (needed for `mysqldump`, `mysql`):

```bash
sudo apt update
sudo apt install -y mysql-client gzip tar
```

Verify tools are available:

```bash
which mysqldump mysql
```

> The NestJS app invokes these binaries via `MYSQLDUMP_PATH`, `MYSQL_PATH`,
> which default to `mysqldump`, `mysql` and resolve from `PATH`. For a VPS you can set absolute paths, e.g. `/usr/bin/mysqldump`.

---

## 2. Environment Configuration

Add the following to `.env`:

```env
# ===== BACKUP & RESTORE SYSTEM =====
BACKUP_ENABLED=true
BACKUP_BASE_DIR=/var/backups/arb
BACKUP_DB_DIR=/var/backups/arb/database
BACKUP_FILE_DIR=/var/backups/arb/files
BACKUP_TMP_DIR=/var/backups/arb/tmp

# Encryption (AES-256-GCM). REQUIRED in production.
# Generate a 32-byte key:  openssl rand -hex 32
BACKUP_ENCRYPTION_KEY=REPLACE_WITH_64_HEX_CHARS
BACKUP_ENCRYPT_ENABLED=true

# pg binaries (absolute paths recommended on VPS)
MYSQLDUMP_PATH=/usr/bin/mysqldump
MYSQL_PATH=/usr/bin/mysql

# Retention policy (defaults)
BACKUP_RETENTION_DAILY_DAYS=7
BACKUP_RETENTION_WEEKLY_WEEKS=8
BACKUP_RETENTION_MONTHLY_MONTHS=12

# Cron schedules
BACKUP_DAILY_CRON=0 2 * * *
BACKUP_WEEKLY_CRON=0 3 * * 0
BACKUP_MONTHLY_CRON=0 4 1 * *
BACKUP_CLEANUP_CRON=0 5 * * *

# Directories to include in file backups (comma separated)
BACKUP_FILE_INCLUDE_DIRS=/var/www/arb/uploads,/var/www/arb/logs

# Compression level (1-9)
BACKUP_COMPRESSION_LEVEL=6
```

---

## 3. Backup Storage Layout

Create the backup directories with restricted permissions:

```bash
sudo mkdir -p /var/backups/arb/{database,files,tmp}
sudo chown -R www-data:www-data /var/backups/arb   # or the app's service user
sudo chmod 700 /var/backups/arb
```

> **SECURITY**: The backup directory is **never served** by Nginx and is not
> part of the public web root. Backup files are encrypted with AES-256-GCM.

To confirm the app user can write, e.g.:

```bash
sudo -u www-data touch /var/backups/arb/tmp/.write-test && sudo -u www-data rm /var/backups/arb/tmp/.write-test
```

---

## 4. Scheduler (NestJS Schedule / Cron)

The NestJS `BackupScheduler` uses `@nestjs/schedule` (in-process cron). It runs
automatically when the app starts:

| Job            | Default Cron      | Description                          |
|----------------|-------------------|--------------------------------------|
| Daily backup   | `0 2 * * *`       | 02:00 daily full backup              |
| Weekly backup  | `0 3 * * 0`       | 03:00 Sunday full backup             |
| Monthly backup | `0 4 1 * *`       | 04:00 1st of month full backup       |
| Cleanup        | `0 5 * * *`       | 05:00 daily retention cleanup        |
| Job processor  | `*/5 * * * *`     | Every 5 min process queued backups   |

Because the app manages cron internally, **no OS-level cron is strictly
required** for scheduled backups. However, for production we recommend an
OS-level `systemd` timer or cron as a **watchdog** to ensure the app is running:

```bash
crontab -e
```

```cron
# Watchdog: every 5 minutes, ensure the app is up (optional)
*/5 * * * * /usr/bin/curl -fsS http://127.0.0.1:3000/health >/dev/null 2>&1 || echo "arb backend down" >> /var/log/arb-watchdog.log
```

---

## 5. Manual Systemd Service (Recommended)

Create `/etc/systemd/system/arb-backend.service`:

```ini
[Unit]
Description=Fasecmo Backend
After=network.target

[Service]
Type=simple
User=www-data
WorkingDirectory=/var/www/arb/backend
EnvironmentFile=/var/www/arb/backend/.env
ExecStart=/usr/bin/node dist/main.js
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target
```

Enable and start:

```bash
sudo systemctl daemon-reload
sudo systemctl enable --now arb-backend
sudo systemctl status arb-backend
```

---

## 6. Running Migrations

Apply the backup schema migration:

```bash
cd /var/www/arb/backend
npx prisma migrate deploy
npx prisma generate
npm run build
sudo systemctl restart arb-backend
```

---

## 7. Security Hardening

- **Encryption**: `BACKUP_ENCRYPT_KEY` must be a 64-char hex string. The app
  fails fast if encryption is enabled but no key is set.
- **File permissions**: `chmod 700` on `/var/backups/arb`; only the app user
  can read/write backups.
- **Never expose backups publicly**: Nginx config must not serve `/var/backups`.
- **Database credentials**: `DATABASE_URL` password is passed to `mysqldump` via
  the `MYSQL_PWD` env var (never on the command line / process list).
- **Secrets excluded**: file backups exclude `.env`, `*.pem`, `*.key`, private
  keys, `node_modules`, `.next`, and `dist` by default.
- **Only Super Admin** can create/restore/delete/download backups (RBAC via
  `@Roles('SUPER_ADMIN')`).

---

## 8. Restore Procedure (Operational)

1. Log in as Super Admin.
2. Open **Super Admin → Backup History**.
3. Locate the backup to restore and click **Restore**.
4. The system verifies the SHA-256 checksum **before** restoring.
5. Database is restored first, then files.
6. Confirm the restore result and any target database override.

> For a full disaster-recovery restore, you may also restore manually:
>
> ```bash
> # Decrypt (if encrypted)
> openssl enc -d -aes-256-gcm -K <key> -iv <iv> -in backup.enc -out backup.gz
> # Restore DB
> gunzip -c backup.gz | mysql "$DATABASE_URL"
> ```

---

## 9. Monitoring & Alerts

- Every backup is **auto-verified** after creation (SHA-256 + decrypt sanity).
- Failed backups/verifications are logged and persisted as `BackupHistory`
  records with `status=FAILED`.
- Check the Admin monitoring dashboard (`/health`, `/monitoring/*`) for DB
  health, and the backup dashboard for retention/status counts.

---

## 10. Off-site / DR (Recommended)

For production DR, sync encrypted backups off-site daily, e.g. with `rclone`:

```bash
# Install rclone and configure a remote (e.g., S3, B2, GCS)
rclone sync /var/backups/arb remote:arb-backups --encrypt-webdav-... \
  --transfers 4 --checksum --create-empty-src-dirs
```

Use a systemd timer or cron to run the off-site sync nightly:

```cron
0 6 * * * /usr/bin/rclone sync /var/backups/arb remote:arb-backups >> /var/log/rclone.log 2>&1
```

---

## 11. Verify Installation

1. Start the app and check logs for `Backup directories ensured`.
2. Trigger a manual backup from the Admin Dashboard.
3. Confirm a `.gz`/`.enc` file appears under `/var/backups/arb/database`.
4. Confirm a `BackupHistory` and a `BackupVerification` (PASSED) record.
5. Run `npm test` from the backend to run the backup unit tests.