# Operations Runbook

## Health and incident triage

1. Request `/api/v1/health` and identify the degraded dependency.
2. Check `docker compose ps` and bounded logs for `web`, `app`, `worker`, and `scheduler`.
3. If database/storage failed, stop mutating traffic before recovery. If queue/scheduler degraded, reads may remain available while workers are repaired.
4. Never paste environment files, tokens, prompts, learner content, or database dumps into tickets or chat.

## Queue and scheduler

- Inspect failures with `php artisan queue:failed`; retry only after fixing the cause.
- Restart workers gracefully using `php artisan queue:restart`.
- `system:heartbeat` runs every minute. A stale heartbeat indicates scheduler or cache failure.
- `exports:prune` removes only expired completed/failed/cancelled artifacts.

## Export and certificate incidents

- Failed exports retain a bounded error and may be retried. MP4 requires a configured render worker and fails honestly when unavailable.
- Never mark an artifact completed unless it exists on the configured disk.
- Certificate issuance is idempotent for learner + Course Version. Revocation requires a reason and retains the immutable snapshot.
- Public verification never exposes email, internal IDs or learning analytics.

## Backup and restore

Run `scripts/backup.sh /backups` with `DB_HOST`, `DB_PORT`, `DB_DATABASE`, `DB_USERNAME`, and `DB_PASSWORD` plus private storage mounted. Copy the timestamped directory to separate encrypted storage and verify both artifacts are non-empty.

Restore first into an isolated target with `scripts/restore.sh /backups/TIMESTAMP`. Verify migrations, health, login, one signed Asset, one export and one certificate before switching traffic. Restore overwrites state and requires an approved maintenance window.

## Upgrade

Read migration notes, take a verified backup, deploy the same image digest to app/worker/scheduler, run migrations once, restart workers, clear caches and perform role/API smoke tests. Keep previous images through the observation window.