2.0 KiB
2.0 KiB
Operations Runbook
Health and incident triage
- Request
/api/v1/healthand identify the degraded dependency. - Check
docker compose psand bounded logs forweb,app,worker, andscheduler. - If database/storage failed, stop mutating traffic before recovery. If queue/scheduler degraded, reads may remain available while workers are repaired.
- Never paste environment files, tokens, prompts, learner content, or database dumps into tickets or chat.
Queue and scheduler
- Inspect failures with
php artisan queue:failed; retry only after fixing the cause. - Restart workers gracefully using
php artisan queue:restart. system:heartbeatruns every minute. A stale heartbeat indicates scheduler or cache failure.exports:pruneremoves only expired completed/failed/cancelled artifacts.
Export and certificate incidents
- Failed exports retain a bounded error and may be retried. MP4 requires a configured render worker and fails honestly when unavailable.
- Never mark an artifact completed unless it exists on the configured disk.
- Certificate issuance is idempotent for learner + Course Version. Revocation requires a reason and retains the immutable snapshot.
- Public verification never exposes email, internal IDs or learning analytics.
Backup and restore
Run scripts/backup.sh /backups with DB_HOST, DB_PORT, DB_DATABASE, DB_USERNAME, and DB_PASSWORD plus private storage mounted. Copy the timestamped directory to separate encrypted storage and verify both artifacts are non-empty.
Restore first into an isolated target with scripts/restore.sh /backups/TIMESTAMP. Verify migrations, health, login, one signed Asset, one export and one certificate before switching traffic. Restore overwrites state and requires an approved maintenance window.
Upgrade
Read migration notes, take a verified backup, deploy the same image digest to app/worker/scheduler, run migrations once, restart workers, clear caches and perform role/API smoke tests. Keep previous images through the observation window.