33 خطوط
2.0 KiB
Markdown
33 خطوط
2.0 KiB
Markdown
# Operations Runbook
|
|
|
|
## Health and incident triage
|
|
|
|
1. Request `/api/v1/health` and identify the degraded dependency.
|
|
2. Check `docker compose ps` and bounded logs for `web`, `app`, `worker`, and `scheduler`.
|
|
3. If database/storage failed, stop mutating traffic before recovery. If queue/scheduler degraded, reads may remain available while workers are repaired.
|
|
4. Never paste environment files, tokens, prompts, learner content, or database dumps into tickets or chat.
|
|
|
|
## Queue and scheduler
|
|
|
|
- Inspect failures with `php artisan queue:failed`; retry only after fixing the cause.
|
|
- Restart workers gracefully using `php artisan queue:restart`.
|
|
- `system:heartbeat` runs every minute. A stale heartbeat indicates scheduler or cache failure.
|
|
- `exports:prune` removes only expired completed/failed/cancelled artifacts.
|
|
|
|
## Export and certificate incidents
|
|
|
|
- Failed exports retain a bounded error and may be retried. MP4 requires a configured render worker and fails honestly when unavailable.
|
|
- Never mark an artifact completed unless it exists on the configured disk.
|
|
- Certificate issuance is idempotent for learner + Course Version. Revocation requires a reason and retains the immutable snapshot.
|
|
- Public verification never exposes email, internal IDs or learning analytics.
|
|
|
|
## Backup and restore
|
|
|
|
Run `scripts/backup.sh /backups` with `DB_HOST`, `DB_PORT`, `DB_DATABASE`, `DB_USERNAME`, and `DB_PASSWORD` plus private storage mounted. Copy the timestamped directory to separate encrypted storage and verify both artifacts are non-empty.
|
|
|
|
Restore first into an isolated target with `scripts/restore.sh /backups/TIMESTAMP`. Verify migrations, health, login, one signed Asset, one export and one certificate before switching traffic. Restore overwrites state and requires an approved maintenance window.
|
|
|
|
## Upgrade
|
|
|
|
Read migration notes, take a verified backup, deploy the same image digest to app/worker/scheduler, run migrations once, restart workers, clear caches and perform role/API smoke tests. Keep previous images through the observation window.
|